Journal of The Korean Association For Science Education (한국과학교육학회지)

The Korean Association for Science Education (한국과학교육학회)
권호 표지
DOI QR코드

DOI QR Code

A School-tailored High School Integrated Science Q&A Chatbot with Sentence-BERT: Development and One-Year Usage Analysis

인공지능 문장 분류 모델 Sentence-BERT 기반 학교 맞춤형 고등학교 통합과학 질문-답변 챗봇 -개발 및 1년간 사용 분석-

  • Gyeongmo Min (Seoul National University) ;
  • Junehee Yoo (Seoul National University)
  • 민경모 (서울대학교 물리교육과) ;
  • 유준희 (서울대학교 물리교육과)
  • Received : 2024.03.27
  • Accepted : 2024.04.24
  • Published : 2024.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study developed a chatbot for first-year high school students, employing open-source software and the Korean Sentence-BERT model for AI-powered document classification. The chatbot utilizes the Sentence-BERT model to find the six most similar Q&A pairs to a student's query and presents them in a carousel format. The initial dataset, built from online resources, was refined and expanded based on student feedback and usability throughout over the operational period. By the end of the 2023 academic year, the chatbot integrated a total of 30,819 datasets and recorded 3,457 student interactions. Analysis revealed students' inclination to use the chatbot when prompted by teachers during classes and primarily during self-study sessions after school, with an average of 2.1 to 2.2 inquiries per session, mostly via mobile phones. Text mining identified student input terms encompassing not only science-related queries but also aspects of school life such as assessment scope. Topic modeling using BERTopic, based on Sentence-BERT, categorized 88% of student questions into 35 topics, shedding light on common student interests. A year-end survey confirmed the efficacy of the carousel format and the chatbot's role in addressing curiosities beyond integrated science learning objectives. This study underscores the importance of developing chatbots tailored for student use in public education and highlights their educational potential through long-term usage analysis.

본 연구에서는 오픈소스 소프트웨어와 인공지능 문서 분류 모델인 한국어 Sentence-BERT로 고등학교 1학년 통합과학 질문-답변 챗봇을 제작하고 2023학년도 1년 동안 독립형 서버에서 운영했다. 챗봇은 Sentence-BERT 모델로 학생의 질문과 가장 유사한 질문-답변 쌍 6개를 찾아 캐러셀 형태로 출력한다. 질문-답변 데이터셋은 인터넷에 공개된 자료를 수집하여 초기 버전을 구축하였고, 챗봇을 1년 동안 운영하면서 학생의 의견과 사용성을 고려하여 자료를 정제하고 새로운 질문-답변 쌍을 추가했다. 2023학년도 말에는 총 30,819개의 데이터셋을 챗봇에 통합하였다. 학생은 챗봇을 1년 동안 총 3,457건 이용했다. 챗봇 사용 기록을 빈도분석 및 시계열 분석한 결과 학생은 수업 중 교사가 챗봇 사용을 유도할 때 챗봇을 이용했고 평소에는 방과 후에 자습하면서 챗봇을 활용했다. 학생은 챗봇에 한 번 접속하여 평균적으로 2.1~2.2회 정도 질문했고, 주로 사용한 기기는 휴대폰이었다. 학생이 챗봇에 입력한 용어를 추출하고자 한국어 형태소 분석기로 명사와 용언을 추출하여 텍스트 마이닝을 진행한 결과 학생은 과학 질문 외에도 시험 범위 등의 학교생활과 관련된 용어를 자주 입력했다. 학생이 챗봇에 자주 물어본 주제를 추출하고자 Sentence-BERT 기반의 BERTopic으로 학생의 질문을 두 차례 범주화하여 토픽 모델링을 진행했다. 전체 질문 중 88%가 35가지 주제로 수렴되었고, 학생이 챗봇에 주로 물어보는 주제를 추출할 수 있었다. 학년말에 학생을 대상으로 한 설문에서 챗봇이 캐러셀 형태로 결과를 출력하는 형태가 학습에 효과적이었고, 통합과학 학습과 학습 목적 이외의 궁금증이나 학교생활과 관련된 물음에 답해주는 역할을 수행했음을 확인할 수 있었다. 본 연구는 공교육 현장에서 학생이 실제로 활용하기에 적합한 챗봇을 개발하여 학생이 장기간에 걸쳐 챗봇을 사용하는 과정에서 얻은 데이터를 분석함으로써 학생의 요구를 충족할 수 있는 챗봇의 교육적 활용 가능성을 확인했다는 점에 의의가 있다.

Keywords

References

  1. Ahn, C., Jeon, H., & Hahn, H. (2023). Implementation of Chatbot Models for Coding Education. The Journal of The Institute of Internet, Broadcasting and Communication, 23(1), 29-35. https://doi.org/10.7236/JIIBC.2023.23.1.29
  2. Akhtar, M., Neidhardt, J., & Werthner, H. (2019, July). The potential of chatbots: analysis of chatbot conversations. In 2019 IEEE 21st conference on business informatics (CBI) (Vol. 1, pp. 397-404). IEEE.
  3. An, J. (2024). bespin-global/klue-sroberta-base-continue-learning-by-mnr. Hugging Face. Retrieved February 16, 2024, from https://huggingface.co/bespin-global/klue-sroberta-base-continue-learning-by-mnr
  4. Bae, J., & Beom, E. (2021). Big data research: writing a thesis with big data. p86. Paju: Jungminsa
  5. Chang, J., Park, J., & Park, J. (2021). An analysis on the trends of education research related to 'Artificial Intelligence chatbot' in Korea: Focusing on implications for use in science education. Korean Association for Learner-Centered Curriculum and Instruction, 21(13), 729-743.
  6. Chin, C., & Osborne, J. (2008). Students' questions: A potential resource for teaching and learning science. Studies in Science Education, 44(1), 1-39.
  7. Choi, S., & Nam, J. (2019). The Use of AI Chatbot as An Assistant Tool for SW Education. Journal of the Korea Institute of Information and Communication Engineering, 23(12), 1693-1699.
  8. Gan, J. (2022). jhgan/ko-sroberta-multitask.Hugging Face. Retrieved February 16, 2024, from https://huggingface.co/jhgan/ko-srobertamultitask
  9. Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. ArXiv Preprint ArXiv:2203.05794.
  10. Hamzah, W. W., Ismail, I., Yusof, M. K., Saany, S. M., & Yacob, A. (2021). Using learning analytics to explore responses from student conversations with chatbot for education. International Journal of Engineering Pedagogy, 11(6), 70-84.
  11. Hwang, H. (2021). Development of Chatbot for Elementary Social Studies Micro-learning. Social Studies Education, 60(3), 81-104
  12. Hwang, H. (2022). The Application of Map AI Chatbot Class for Micro-learning in Social Studies. Social Studies Education, 61(2), 69-94.
  13. Jang, J., Park, J., & Park. S. (2021). An analysis on the trends of education research related to 'Artificial Intelligence chatbot' in Korea: Focusing on implications for use in science education. Korean Association for Learner-Centered Curriculum and Instruction, 21(13), 729-743.
  14. Jeong, S., Lee, S., Kwon, D., Kim, J., & Jung, E. (2023). Design and Implementation of an Educational Resource Recommendation System for Programming Languages. In Proceedings of KIIT Conference (pp. 373-375).
  15. Jho, H. (2023). Understanding of Generative Artificial Intelligence Based on Textual Data and Discussion for Its Application in Science Education. Journal of the Korean Association for Science Education, 43(3), 307-319.
  16. Kang, Y., & Lee, M. (2021). Convergence of a Classical Novel and Computing Thinking through the Development of a Tokkijeon Chatbot - A Gamification Chatbot on an Open Source Chatbot Platform -. The Studies of Korean Literature, 70, 29-64.
  17. Kasneci, E., Sessler, K., Kuchemann, S., Bannert, M., Dementieva, D., Fischer, F., ... & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274.
  18. Kim, J., & Lee, M. (2021). Research and Development of a Hong Gil-Dong Jeon Chatbot Prototype based on a Chatbot Builder - Focusing on Fusion Education in Classical Novels and Computational Thinking -. International Association of Language and Literature, 88, 47-8
  19. Kim, M., & Huh, S. (2021). The Development of a Web-based Learning Support Tool Prototype Using AI Chatbot in Coding Education. Journal of Korean Association for Educational Information and Media, 27(1), 111-132.
  20. Kim, M., Yeom, J., Jung, H., & Lim, C. (2021). A Review of Research on Artificial Intelligence Chatbot in Education through the Lens of Activity Theory. The Journal of Educational Information and Media, 27(2), 699-721.
  21. Kim, S., & Kim, Y. (2023). Analysis of AI Education Research Trends in Korea using LDA, BERTopic. Proceedings of Korean Institute of Information Scientists and Engineers, 772-774.
  22. Kim, S., Yoon, T, & Kang, J. (2023). A Study on the Clustering of Dispute Types through Judicial Precedent Analysis Related to 'Elderly' Based on BERTopic and Social Network Analysis. The Journal of Society for e-Business Studies, 28(1), 123-144.
  23. Ko, Y., Lee, S., Cha, M., Kim, S., Lee, J., Han, J., & Song, M. (2022). Topic Modeling Insomnia Social Media Corpus using BERTopic and Building Automatic Deep Learning Classification Model. Korean Society for Information Management, 39(2), 111-129.
  24. Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973-1018.
  25. Kwon, Y. (2019). Education Data and Analytics: A Review of the State of the Art. The Korea Journal of BigData, 4(1), 73-81.
  26. Lee, M. (2022). Kiwi, Korean Intelligent Word Identifier. In GitHub repository (v0.14.0) [Computer software]. GitHub. https://doi.org/10.5281/zenodo.7041425
  27. Lee, S., & Song, K. (2023). Prompt engineering to improve the performance of teaching and learning materials Recommendation of Generative Artificial Intelligence. Journal of The Korea Society of Computer and Information, 28(8), 195-204.
  28. Lee. Y. (2020). A Study on Carousel Design UX. Journal of Digital Convergence, 18(12).
  29. Li, C., Chen, K., & Chang, Y. (2019). When there is no progress with a task-oriented chatbot: A conversation analysis. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 1-6).
  30. Min, G., & Yoo, J. (2022). Development of a middle school science Q&A chatbot using Doc2Vec and analysis of student's queries. The SNU Journal of Education Research, 31(3), 115-145.
  31. Mo, Y. (2020). A Developmental Study on a Chatbot for Digital Storytelling. Master's thesis. Seoul National University, Seoul, South Korea.
  32. Mondal, A., Dey, M., Das, D., Nagpal, S., & Garda, K. (2018). Chatbot: An automated conversation system for the educational domain. 2018 International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP).
  33. Park, J., Kim, D., & Yoo, S. (2020). Chatbot system design for extracurricular education program inquiry using BERT model. The Korean Institute of Information Scientists and Engineers, 1596-1598.
  34. Park, S., & Kang, J. (2023). The complete guide to text mining in Python : From natural language processing basics to deep learning-based BERT and transformers (Revised ed.). Paju: Wikibooks.
  35. Peyton, K., & Unnikrishnan, S. (2023). A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs. Results in Engineering, 17, 100856.
  36. Raamadhurai, S., Baker, R., & Poduval, V. (2019). Curio SmartChat: A system for natural language question answering for self-paced K-12 learning. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 336-342).
  37. Shin, W., & Shin, D. (2020). A Study on the Application of Artificial Intelligence in Elementary Science Education. Journal of Korean Elementary Science Education, 39(1), 117-132.
  38. Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151.
  39. Sousa, E. B. D., Alexandre, B., Ferreira Mello, R., Pontual Falcao, T., Vesin, B., & Gasevic, D. (2021). Applications of learning analytics in high schools: A systematic literature review. Frontiers in Artificial Intelligence, 4, 737891.
  40. Yang, H., & Kim, K. (2023). Development of a ChatGPT-Based Chabot for Science Education in 6th Grade Elementary School. Journal of The Korean Association of Information Education, 27(6), 729-739.
  41. Yoo, W., & An, S. (2022). 18-08 Korean chatbot using BERT's sentence embedding (SBERT). Retrieved February 16, 2024, from https://wikidocs.net/154530