The Korean Journal of Emergency Medical Services (한국응급구조학회지)

The Korea Society of Emergency Medical Services (한국응급구조학회)
권호 표지
DOI QR코드

DOI QR Code

A study of the adequate number of questions in a mock test for the paramedic national examination using item response theory

문항반응이론을 적용한 1급 응급구조사 국가시험 대비 모의시험의 적정성 연구

  • Jung Eun Lee (Department of Paramedicin, Dongnam Health University) ;
  • Jundong Moon (Department of emergency medical service, Kongju National University) ;
  • Ajung Kim (Depatment of Paramedicine, Kyungil University)
  • 이정은 (동남보건대학 응급구조과) ;
  • 문준동 (국립공주대학교 응급구조학과) ;
  • 김아정 (경일대학교 응급구조학과)
  • Received : 2024.06.15
  • Accepted : 2024.08.13
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Purpose: To adjust item numbers in a national test, this study used item response theory to examine changes in average scores, reliability, difficulty, and discrimination according to the adjustment of item numbers. Methods: We analyzed the dichotomous coding of correct and incorrect answers of 473 examinees in a mock test conducted in 2023. Additionally, as an explanatory pilot study, we used an online questionnaire to survey experts on their perceptions of the appropriate item numbers for each test subject from January 18, 2024, to February 15, 2024. Results: Regarding the item numbers on the national exam, experts preferred to reduce the number of management of emergency patients (33.14±6.09, p<.05) and advanced emergency medical care: subtopics (104.49±11.55, p<.05), and the total number of questions (217.82±20.95, p<.05). In a simulation set in which items with low item fit were removed after fitting a two-parameter item response theory model, reliability was maintained at .910 until the 5th test consisting of 185 questions with little loss of difficulty, discrimination, and average score, and there was no correlation between item numbers and average score. Conclusion: Experts responded that reducing the number of items on the national exam was appropriate. As a result of the item reduction simulation, there was no significant loss in the average score, difficulty, discrimination, or reliability. More reliable results could be obtained if the results were based on a validity analysis and analyzed using actual national exams.

Keywords

Acknowledgement

본 연구는 동남보건대학교 연구비 지원에 의해 수행된 것이다.

References

  1. Emergency medical service act. Available at https://www.law.go.kr/LSW/eng/engLsSc.do?menuId=2§ion=lawNm&query=emergency&x=47&y=26#liBgcolor4
  2. Korea health personnel incensing examination institute. National exam analysis results. Available at https://www.kuksiwon.or.kr/analysis/brd/m_91/list.do
  3. Korea health personnel incensing examination institute. Health professional categories. Available at https://www.kuksiwon.or.kr/cnt/c_1/view.do?seq=7
  4. Lee SE, Park YJ, Lee DJ. Curriculum and educational evaluation. KNOU press, 2022. 290-341.
  5. Yim CI, Yim JH, Lee DJ. Educational technology. KNOU press, 2022. 249-50.
  6. Kim MY. Development of practice-based integrated question type for national nursing examination. Korea Health Personnel Licensing Examination institute, 2023.
  7. Sung TJ, Kim KH. Changes in reliability coefficient and test information function according to changes in number of items, item difficulty, and item discrimination. J Curric Eval 1993;6(2):123-54.
  8. Yim MK, Huh S. Testing unidimensionality and goodness-of-fitness for the application of item response theory to the Korean Medical Licensing Examination. Korean J Med Educ 2007;19(2):163-9. http://doi.org/10.3946/kjme.2007.19.2.163
  9. Jeong GH, Yim MK. Applicability of item response theory to the Korean Nurses' Licensing Examination. J Educ Eval Health Prof 2005;22(1):23-9. http://doi.org/ 10.3352/jeehp.2005.2.1.23
  10. Kim AJ, Park TJ, Bak YS, Kim JH, Kim YS, Son YM et al. Review of the qualification criteria for the national examinations for emergency medical technicians. Korean J Emerg Med Ser 2021;25(2):39-53. http://doi.org/10.14408/KJEMS.2021.25.2.039
  11. Kim MY, Lee YH, Huh S. Correlations between the scores of computerized adaptive testing, paper and pencil tests, and the Korean Medical Licensing Examination. J Educ Eval Health Prof 2005;2(1):113-8. http://doi.org/10.3352/jeehp.2005.2.1.113
  12. Yoou SK, Jo JM, Kim JH, Uhm TH. Proposal of a new EMT national registry exam using the Delphi method. Korean J Emerg Med Ser 2010;14(2):57-69.
  13. Yoou SK, Choi ES, Shin DM, Jo JM, Lee JE, Roh SG et al. Proposal of a new emergency medical technician national practical examination. Korean J Emerg Med Ser 2012;16(3):45-62 (Korean). http://doi.org/10.14408/KJEMS.2012.16.3.045
  14. Park HG, Kim DW, Kang YJ, Bae SY. Analysis of the appropriate number of questions that satisfy the validity and confidence level of the test. Korea Health Personnel Licensing Examination Institute, 2022.
  15. Kim YH, Hang MJ. Trend analysis of the results of the national examination for EMT-paramedics for the Past 9 Years. Journal of Health Informatics and Statistics 2022;47(2):133-8. http://doi.org/10.21032/jhis.2022.47.2.133
  16. Brogden HE. Variation in test validity with variation in the distribution of item difficulties, number of items, and degree of their intercorrelation. Psychometrika 1946;11(4):197-214. https://doi.org/10.1007/BF02290130
  17. Baiju RM. Academic stress and psychological well-being among college students. Available at: https://ijip.in/wp-content/uploads/2021/08/18.01.022.0210903.pdf. Accessed May 18, 2024.
  18. Suruchi S, Rana SS. Test item analysis and relationship between difficulty level and discrimination index of test items in an achievement test in biology. PIJR. 2014;3(6):56-8. https://doi.org/10.15373/22501991/June2014/18
  19. Smolen T, Chuderski A. The quadratic relationship between difficulty of intelligence test items and their correlations with working memory. Front Psychol. 2015;6:156499.
  20. Stankovska G, Dimitrovski D, Angelkoska S, Ibraimi Z, Uka V. Emotional intelligence, test anxiety and academic stress among university students. Bulgarian Comparative Education Society. 2018. Available at: http://files.eric.ed.gov/fulltext/ED586176.pdf. Accessed May 18, 2024.
  21. Kaur M, Singla S, Mahajan R. Item analysis of in use multiple choice questions in pharmacology. Int J Appl Basic Med Res. 2016;6(3):170-3. https://doi.org/10.4103/2229-516x.186965
  22. Kim MY. Development of practice-based integrated question type for national nursing examination. Korea Health Personnel Licensing Examination Institute. 2023.
  23. Seong TJ. Educational evaluation. Seoul: Haksisa, 2019. 259-64.
  24. Seong TJ. Understanding and applying item response theory. Paju: Kyoyookbook 2019. 30-4, 56-7.
  25. Columbia University Mailman School of Public Health. Item response theory. Available at: https://www.publichealth.columbia.edu/research/population-health-methods/item-response-theory. Accessed May 18, 2024.
  26. Lim EY, Park JH, Kwon I, Song GL, Huh S. Comparison of item analysis results of Korean Medical Licensing Examination according to classical test theory and item response theory. J Educ Eval Health Prof 2004;1(1):67-76. http://doi.org/10.3352/jeehp.2004.1.1.67