Abstract
Good assessment is crucial for feedback on curriculum and to motivate students to learn. This study was conducted to perform item analysis on the Medical Education Assessment Consortium clinical knowledge mock examination (MEAC CKME) (2011-2016) and to evaluate several effects to improve item quality using both classical test theory and item response theory. The estimated difficulty index (P) and discrimination index (D) were calculated according to each course, item type, A (single best answer)/R (extended matching) type, and grading of item quality. The cut-off values used to evaluate P were: >0.8 (easy); 0.6-0.8 (moderate); and <0.6 (difficult). The cut-off value for D was 0.3. The proportion of appropriate items was defined as those with P between 0.25-0.75 and D ${\geq}0.25$. Cronbach ${\alpha}$ was used to assess the reliability and was compared with those of the Korean Medical Licensing Examination (KMLE). The results showed the recent mean difficulty and decimation index was 0.62 and 0.20 for the first MEAC CKME and 0.71 and 0.19 for the second MEAC CKME, respectively. Higher grade items evaluated by a self-checklist system had better D values than lower grade items and higher grade items gradually increased. The preview and editing process by experts revealed maintained P, decreased recall items, increased appropriate items with better D values, and higher reliability. In conclusion, the MEAC CKME (2011-2016) is deemed appropriate as an assessment to evaluate students' competence and prepare year four medical students for the KMLE. In addition, the self-checklist system for writing good items was useful in improving item quality.
