Journal of Practical Engineering Education (실천공학교육논문지)

Korean Institute for Pratical Engineering Education (한국실천공학교육학회)
권호 표지
DOI QR코드

DOI QR Code

An Exploratory Study on Developing the AI Essay Test Tool based on ChatGPT: Focusing on the Interaction with the Engineer

ChatGPT를 활용한 AI 글쓰기 의사소통 역량 평가도구 개발 과정에 대한 연구: 기술 전문가와의 상호소통을 중심으로

  • So-Young Park (Division of Education, Sookmyung Women's University) ;
  • ByungYoon Lee (Education Research Institute, Sookmyung Women's University, Sookmyung Women's University) ;
  • Yujung Hong (Department of Ethics Education, Seoul National University)
  • 박소영 (숙명여자대학교 교육학부) ;
  • 이병윤 (숙명여자대학교 교육연구소) ;
  • 홍유정 (서울대학교 윤리교육과)
  • Received : 2023.12.15
  • Accepted : 2024.01.29
  • Published : 2024.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

This study focused on the development of an AI essay tool for assessing writing-communication competence using ChatGPT. During the development process, the interaction between content expert and technical expert was emphasized to explore the fusion of IT and humanities and social sciences. Through close communication and interaction between the content and technical experts, they incorporated scoring criteria for writing-communication competence and developed an AI essay test tool that provides scores and feedback in the appropriateness of content, effectiveness of organization, and accuracy of grammar. This process revealed how content and technology combine and presented considerations for future fusion researchers, including technical experts in generative AI assessment tools.

본 연구는 ChatGPT를 활용한 글쓰기 의사소통 역량을 평가하는 AI 평가도구 개발에 초점을 두었다. 개발 과정에 있어 내용 전문가와 기술 전문가 간의 상호작용을 통해 IT와 인문사회 영역의 융합과정을 탐구하였다. 내용 전문가와 기술 전문가는 긴밀한 소통 및 상호작용을 통해, 글쓰기 의사소통 역량의 채점기준을 탑재하였고 각 영역별(내용의 적절성, 조직의 효과성, 어법의 정확성) 점수와 피드백을 제공하는 AI 평가도구를 개발하였다. 이 과정에서 본 연구는 내용과 기술이 어떻게 결합하는지에 대한 과정을 드러내고, 이후 생성형 AI를 활용한 평가도구 기술 전문가를 포함한 융합연구자들이 유의해야 할 사항 등에 대해 제시하였다.

Keywords

Acknowledgement

이 논문은 2020년 대한민국 교육부와 한국연구재단의 지원을 받아 수행된 연구임(NRF-2020S1A3A2A02095447). 이 연구를 수행하기 위해 많은 전문가들의 협업이 필요하였음. 이를 위해 애써준 내용 전문가 여러분과 평가도구를 빠른 시일 내에 개발하기 위해 애써준 김한길 엔지니어께도 감사를 표함.

References

  1. S. H. Kim, K. S. Kim, S. Kim, Y. Kim, J. H. Kim, J. B. Kim, H. I. Kim, H. B. Kim, H. C. Kim, H. R. Kim, D. Park, S. J. Park, C. S. Park, W. S. Sohn, S. Song, J. K. Shim, J. H. Lee, J. S. Lee, S. Jun, W. Jun, Y. Jeong, U. Jung, and J. Seo, "Analysis of functions and services for the development of AI education platform," The Journal of Korean Association of Computer Education, vol. 24, no. 2, pp. 25-37, March, 2021. https://doi.org/10.14352/jkaie.2020.24.4.337
  2. AIEDAP. Monthly Report [Internet]. Available: https://aiedap.or.kr/
  3. H. Jang and H. J. So, "The analysis of research trends and topics about the educational use of ChatGPT," Journal of Research in Curriculum & Instruction, vol. 27, no. 4, pp. 387-401, August, 2023.
  4. AI Times, "AI trends identified at the Edutech Korea Fair," Sep. 22, 2023 [Online]. Available: https://www.aitimes.com/news/articleView.html?idxno=153859
  5. H. Park, S. S. Kim, K. H. Kim, M. Lee, K. K. Kim, and J. Kim, "Substanitializing methods of restricted and extended response essay assessment through enforcing the instruction-assessment alignment," Jincheon: Chungbuk, KICE Report, 2019.
  6. K. Ercikan and D. F. McCaffrey, "Optimizing implementation of artificial intelligence-based automated scoring: an evidence centered design approach for designing assessments for AI-based scoring," Journal of Educational Measurement, vol. 59, no. 3, pp. 272-287, September, 2022. https://doi.org/10.1111/jedm.12332
  7. Korea Education Culture News, "Siwon School Lab-Data Bank, providing AI mock test platform 'The Test Glider'," Oct. 18, 2023 [Online]. Available: https://www.kecn.co.kr/news/articleView.html?idxno=1723
  8. S. W. Han and W. J. Kim, "A study on the effects of communicative competence on information literacy of undergraduates," Journal of the Korean Library and Information Science, vol. 50, no. 1, pp. 377-394, February, 2016. https://doi.org/10.4275/KSLIS.2016.50.1.377
  9. C. Lim, "The development and effects of design and implementation strategies for supporting web-based self-regulated learning," Journal of Educational Technology, vol. 18, no. 4, pp. 3-23, December, 2002. https://doi.org/10.17232/KSET.18.4.3
  10. B. Kim, Y. Jeon, J. Kim, C. Hong, and T. Kim, "A study of IT convergence desired talent based on computation thinking," in Proceeding of the 2014 Winter Conference of the Korean Association of Computer Education, Seoul, pp. 27-33, 2014.
  11. J. M. Lee, T. Hur, J. B. Lee, G. Hwang, and K. Om, "Analysis of human resources practices and career path movement in the field of convergence technologies," The Journal of the Korea Contents Association, vol. 10, no. 5, pp. 446-459, April, 2010. https://doi.org/10.5392/JKCA.2010.10.5.446
  12. J. Kim, "BSC (Balanced Scorecard) implementation factors and firm performance: The role of communication, strategic alignment, and cooperation," Korean Accounting Journal, vol. 20, no. 1, pp. 233-264, March, 2011.
  13. J. Bae and G. Jeon, "A study on the influencing factors and performance of communication patterns among partners in collaborative R&D projects," in Proceeding of the 1998 Spring Conference of KIEE and KORMS, Pusan, pp. 1-9, 1998.
  14. V. Spandel and R. Culham, "Writing assessment," in A Handbook for Student Performance Assessment in an Era of Restructuring. Alexandria, VA: ASCD.
  15. J. Park and Y. Park, "A study on the differences of the anchor papers according the rater consistence," Korean Writing Association, vol. 14, pp. 301-338, March, 2012.
  16. I. Park and B. Kim, "Determinants of successful R&D cooperations between SMEs and public research institutes in Korea," Journal of Korea Technology Innovation Society, vol. 15, no. 4, pp. 783-814, December, 2012.
  17. B. Min, Y. Oh, S. Lee, S. Ahn, K. Kim, M. Kim, J. Son, J. Lee, and S. Chang, "Development of a writing ability diagnosis system for university first-year students - The Seoul National University essay examination," The Korean Journal of Literacy Research, vol. 13, no. 6, pp. 45-76, December, 2022. https://doi.org/10.37736/kjlr.2022.04.13.2.02
  18. Y. M. Park, "The method and procedure of evaluating writing ability," The Journal of Korean Language and Literature Education, pp. 1-29.
  19. S. Seo, "Study on setting the components of writing assignment," Korean Language Education Research, vol. 33, pp. 449-472, December, 2008. https://doi.org/10.20880/kler.2008..33.449
  20. H. Song, "An analysis of basic grammar evaluations for college students' writing," The Journal of Korean Language and Literature Education, vol. 35, pp. 31-57, June, 2015.  https://doi.org/10.17313/jkorle.2015..35.31