The Journal of Korean Medicine (대한한의학회지)

The Society of Korean Medicine (대한한의학회)
권호 표지
DOI QR코드

DOI QR Code

Pilot Development of a 'Clinical Performance Examination (CPX) Practicing Chatbot' Utilizing Prompt Engineering

프롬프트 엔지니어링(Prompt Engineering)을 활용한 '진료수행시험 연습용 챗봇(CPX Practicing Chatbot)' 시범 개발

  • Jundong Kim (Department of Physiology, College of Korean Medicine, Gachon University) ;
  • Hye-Yoon Lee (Division of Humanities and Social Medicine, School of Korean Medicine, Pusan National University) ;
  • Ji-Hwan Kim (Department of Sasang Constitutional Medicine, College of Korean Medicine, Gachon University) ;
  • Chang-Eop Kim (Department of Physiology, College of Korean Medicine, Gachon University)
  • 김준동 (가천대학교 한의과대학 생리학교실) ;
  • 이혜윤 (부산대학교 한의학전문대학원 인문사회의학교실) ;
  • 김지환 (가천대학교 한의과대학 사상체질의학과) ;
  • 김창업 (가천대학교 한의과대학 생리학교실)
  • Received : 2024.02.05
  • Accepted : 2024.02.16
  • Published : 2024.03.01
Download PDF
⟨ Previous Next ⟩

Abstract

Objectives: In the context of competency-based education emphasized in Korean Medicine, this study aimed to develop a pilot version of a CPX (Clinical Performance Examination) Practicing Chatbot utilizing large language models with prompt engineering. Methods: A standardized patient scenario was acquired from the National Institute of Korean Medicine and transformed into text format. Prompt engineering was then conducted using role prompting and few-shot prompting techniques. The GPT-4 API was employed, and a web application was created using the gradio package. An internal evaluation criterion was established for the quantitative assessment of the chatbot's performance. Results: The chatbot was implemented and evaluated based on the internal evaluation criterion. It demonstrated relatively high correctness and compliance. However, there is a need for improvement in confidentiality and naturalness. Conclusions: This study successfully piloted the CPX Practicing Chatbot, revealing the potential for developing educational models using AI technology in the field of Korean Medicine. Additionally, it identified limitations and provided insights for future developmental directions.

Keywords

Acknowledgement

이 논문은 2022년도 가천대학교 교내연구비 지원에 의한 결과임. (GCU-202206720001)

References

  1. Gwak D, Kim M, Kwon J, Yin C. Trends in Korean Medical education research from 2003 to 2022. The Journal of Korean Medical History. 2023;36:89-98. 
  2. Jeong S-H, Kim J-P, Kang Y-J, Jeong HI, Kim KH. A Survey of Recognitions and Satisfaction with Education in Traditional Korean Medicine. Journal of Society of Preventive Korean Medicine. 2020;24:49-56. 
  3. Cho E, Kim Y, Ha J, Park J, Jung H-J. Developing a manual for clinical practice on real patients in Korean Medicine. J Kor Med Edu. 2023;1:15-22. 
  4. Han SY, Lee SH, Chae H. Developing a best practice framework for clinical competency education in the traditional East-Asian medicine curriculum. BMC Med Educ. 2022;22. 
  5. Lim C, Han H, Hong J, Kang Y. 2016 Competency Modeling for Doctor of Korean Medicine & Application Plans. Journal of Korean Medicine. 2016;37:101-13.  https://doi.org/10.13048/jkm.16010
  6. Han C-Y, Kang D-W, Park J-G, Kim B-H, Kim K-S, Kim Y-B, et al. An Analysis of Clerkship Satisfaction in College of Korean Medicine : Focusing on Doctor-patient Role-play and mock CPX. J Korean Med Ophthalmol Otolaryngol Dermatol. 2020;33:12-24. 
  7. Jo H, Min S. The current status and future operations of Clinical Performance Evaluation (CPX) in the nationwide colleges (graduate schools) of Traditional Korean Medicine. The Journal of Korean Medical History. 2020;33:9-21.  https://doi.org/10.15521/JKMH.2020.33.2.009
  8. Jo H-J, Roh J-D, Sung HK, Park J-S. A Survey on Students' Perception of Clinical Performance Examination (CPX) in College of Korean Medicine Using Student Standardized Patients. Journal of Society of Preventive Korean Medicine. 2020;24:1-13. 
  9. Park S, Kim C. Enhancing Korean Medicine Education with Large Language Models: Focusing on the Development of Educational Artificial Intelligence. Journal of Physiology & Pathology in Korean Medicine. 2023;37:134-8.  https://doi.org/10.15188/kjopp.2023.10.37.5.134
  10. Chen B, Zhang Z, Langrene N, Zhu S. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. 2023. 
  11. Renze M, Guven E. The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models. 2024. 
  12. Institute of Medical Education and Training. Revised Clinical Diagnosis. 2nd edition. Seoul: Seoul National University Press; 2015. 
  13. Bae H, Lee S, Lee CY, Kim CE. A Novel Framework for Understanding the Pattern Identification of Traditional Asian Medicine From the Machine Learning Perspective. Front Med (Lausanne). 2022;8:763533. 
  14. Jang D, Yun T-R, Lee C-Y, Kwon Y-K, Kim C-E. GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors. PLOS Digital Health. 2023;2:e0000416. 
  15. Banerjee D, Singh P, Avadhanam A, Srivastava S. Benchmarking LLM powered Chatbots: Methods and Metrics. 2023.