Korean journal of medical education

The Korean Society of Medical Education (한국의학교육학회)
권호 표지
DOI QR코드

DOI QR Code

Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination

  • Yim, Mikyoung (Research and Development Division, Korea Health Personnel Licensing Examination Institute)
  • Received : 2018.01.18
  • Accepted : 2018.09.11
  • Published : 2018.12.01
⟨ Previous Next ⟩

Abstract

Purpose: The purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE. Methods: The process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed. Results: The modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen's ${\kappa}$ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists' survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method. Conclusion: The modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.

Keywords

References

  1. Cizek GJ, Bunch MB. Standard setting: a guide to establishing and evaluating performance standards on tests. Thousand Oaks, USA: Sage Publication Inc.; 2007.
  2. Park H. Clinical skills assessment in Korean Medical Licensing Examination. Korean J Med Educ. 2008;20(4):309-312. https://doi.org/10.3946/kjme.2008.20.4.309
  3. Shim JS, Kim YJ, Kim JA, et al. A study of standard setting for Korea dentist clinical skill test: RE02-1411-02. Seoul, Korea: Korea Health Personnel Licensing Examination Institute; 2014.
  4. Shin SJ, Kim YK, Suh SR, Jung DY, Kim YJ, Yim MK. A study of adoption of clinical skill test as Korea nurse licensing examination: RE02-1407-05. Seoul, Korea: Korea Health Personnel Licensing Examination institute;2014.
  5. Ahn DS, Im H. Standard setting in student assessment by criterion referenced evaluation. Korean J Med Educ. 2001;13(1):41-45.
  6. Lee G. A psychometric approach to setting a passing score on Korean National Medical Licensing Examination. J Educ Eval Health Prof. 2004;1(1):5-14. https://doi.org/10.3352/jeehp.2004.1.1.5
  7. Jager RM. Certification of student competence. In: Linn RL, ed. Educational Measurement. 3rd ed. New York, USA: American Council on Education; 1989:485-514.
  8. Clauser BE, Margolis MJ, Case SM. Testing for licensure and certification in the professions. In: Brennan RL, ed. Educational Measurement. 4th ed. Westport, USA: Praeger Publishers; 2006:701-731.
  9. Angoff WH. Scales, norms and equivalent scores. In: Thorndike RL, Angoff WH, American Council on Education, eds. Educational Measurement. 2nd ed. Washington, USA: American Council on Education;1971:508-600.
  10. Mitzel HC, Lewis DM, Patz RJ, Green DR. The bookmark procedure: psychological perspectives. In: Cizek GJ, ed. Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, USA: Lawrence Erlbaum Associates; 2001:249-281.
  11. Buckendahl CW, Ferdous AA, Gerrow J. Recommending cut scores with a subset of items: an empirical illustration. Pract Assess Res Eval. 2010;15(6):1-10.
  12. Peterson CH, Schulz EM, Engelhard Jr G. Reliability and validity of bookmark-based methods for standard setting: comparisons to Angoff-based methods in the National Assessment of Educational Progress. Educ Meas Issues Pract. 2011;30(2):3-14.
  13. Kim TE. The comparison of applicability: Angoff and bookmark methods in standard setting [dissertation]. Seoul, Korea: Ewha Womans University; 2004.
  14. Kim NJ. The standard setting of clinical performance examination (CPX) by modified-Angoff, bookmark, and item-descriptor matching (IDM) Method [dissertation]. Seoul, Korea: Ewha Womans University; 2010.
  15. Cetin S, Gelbal S. A comparison of bookmark and Angoff standard setting methods. Educ Sci Theory Pract. 2013;13(4):2169-2175.
  16. Yin P, Schulz EM. A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standard setting. Paper presented at: the annual meeting of the National Council on Measurement in Education; April 12-14, 2005; Montreal, Canada.
  17. Kane MT. So much remains the same: conception and status of validation in standard setting. In: Cizek GJ, ed. Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, USA: Lawrence Erlbaum Associates; 2001:53-88.
  18. Zimowski MF, Muraki E, Mislevy RJ, Bock RD. BILOG-MG3.0. Skokie, USA: Scientific Software International; 2003.
  19. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83(8):761-767. https://doi.org/10.1097/ACM.0b013e31817eb632
  20. Ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):669-675. https://doi.org/10.3109/0142159X.2010.500897
  21. Hambleton RK. Setting performance standards on educational assessments and criteria for evaluating the process. In: Cizek GJ, ed. Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, USA: Lawrence Erlbaum Associates; 2001:89-116.
  22. Karantonis A, Sireci SG. The bookmark standard-setting method: a literature review. Educ Meas Issues Pract. 2006;25(1):4-12.
  23. Jang YS, Seong TJ. The comparison for IRT based standard setting methods: bookmark, IDM and mapmark. J Educ Eval. 2009;22:659-680.

Cited by

  1. Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee vol.15, pp.None, 2018, https://doi.org/10.3352/jeehp.2018.15.32
  2. The Use of Rasch and Item Mapping in Determining Cut Score of Comprehensive Pre internship Exam vol.11, pp.3, 2018, https://doi.org/10.29252/rme.11.3.59
  3. Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination vol.17, pp.None, 2018, https://doi.org/10.3352/jeehp.2020.17.14
  4. Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination vol.17, pp.None, 2020, https://doi.org/10.3352/jeehp.2020.17.28
  5. Multiple‐Choice versus Open‐Ended Questions in Advanced Clinical Neuroanatomy: Using a National Neuroanatomy Assessment to Investigate Variability in Performance Using Different Question vol.14, pp.3, 2021, https://doi.org/10.1002/ase.2053
  6. Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model vol.16, pp.11, 2018, https://doi.org/10.1371/journal.pone.0257871