• Journal of Korean language education
• /
• v.29 no.1
• /
• pp.85-108
• /
• 2018

This study pointed out the reality that many novice Korean language teachers who lack rater training are scoring the learners' writing skill. The study performed and analyzed a case where pre-service teachers were educated in order to explore the possibility of promoting rater training in a Korean language teacher training course. The pre-service teachers majoring in Korean language education at the graduate school scored TOPIK compositions and were provided feedback by the FACETS program, which were further discussed at the rater meeting. In three scoring processes, the raters scored with conscious of own rating patterns and showed positive change or over correction due to excessive consciousness. Consequentially, ongoing training can improve rating ability, and considering the fact that professional rater training is hard to progress, the method composed of FACETS analysis and rater training revealed positive effects. On the other hand, the rater training including native Korean and non-native(Chinese) speakers together showed no significant difference by mother tongue but by individual difference. This can be interpreted as a positive implication to the rating reliability of non-native speakers possessing advanced Korean language abilities. However, this must be supplemented through extended research.

• English Language & Literature Teaching
• /
• v.16 no.1
• /
• pp.123-142
• /
• 2009

The present study used multi-faceted Rasch measurement to explore the characteristics and bias patterns of non-native raters when they scored L2 writing tasks. Three raters scored 254 writing tasks written by Korean university students on two topics adapted from the TOEFL Test of Written English (TWE). The written products were assessed using a five-category rating scale (Content, Organization, Language in Use, Grammar, and Mechanics). The raters only showed a difference in severity with regard to rating categories but not in task types. Overall, the raters scored Grammar most harshly and Organization most leniently. The results also indicated several bias patterns of ratings with regard to the rating categories and task types. In rater-task bias interactions, each rater showed recurring bias patterns in their rating between two writing tasks. Analysis of rater-category bias interaction showed that the three raters revealed biased patterns across all the rating categories though they were relatively consistent in their rating. The study has implications for the importance of rater training and task selection in L2 writing assessment.

• KIEAE Journal
• /
• v.12 no.3
• /
• pp.69-75
• /
• 2012

Developing reliable measures of the environment is important to increase our understanding of the environmental effects on walking among seniors. As a preliminary study for developing an instrument for measuring walkability of seniors' environment, the purpose of this study are to identify important assessment indicators associated with seniors' walking and to test their reliability using inter-rater and intra-rater reliability methods. A set of assessment indicators was identified through literature review, and field studies by trained raters were conducted in three senior centers located in Seongnam area in order to test reliability of the audit tool. The results indicated high percent agreement for most indicators and overall 91.6% and 86.1% of items assessed had good or medium inter-rater and intra-rater reliability, respectively. The reliable assessment indicators would provide reliable data for use in community-based audits of built environment in relation to walking among older adults. The findings showed that the indicators of aesthetics had lower reliability compare to safety, convenience, and access. Rater training with various images would improve rater agreement while reduce rater bias.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.10 no.9
• /
• pp.2424-2434
• /
• 2009

This study, centering on public enterprises, has been conducted from 4 different perspectives such as grasping the effects that the factor precedent of the multi-rater feedback affects on the usefulness of evaluation and the effects that the usefulness of evaluation affects on the organizational achievement, checking on whether there is any discrepancy among the effects affecting on the usefulness of evaluation in accordance with the degree of advance training on the evaluation system, and being what kind of effects the usefulness of valuation that the appraised eventually understand affects on the organizational achievement. As result of the study, the factor precedent that are confidence of the appraisers and the support from CEO reasonably affected on the usefulness of evaluation, but the standard referred to evaluation was not reasonable. It became resultantly clear that the confidence of the appraisers held the effect of control from the verification of whether there is the effect of control between the factor precedent and the usefulness of evaluation in accordance with the degree of advance training. However, it turned out that the standard referred to evaluation and the support from CEO held no effect of control. As result, It was possible to confirm that the usefulness of evaluation in this study affects directly or indirectly to the organizational achievement such as organizational immersion, job satisfaction, self development opportunity, and organizational revitalization as influenced by the prior factors, and the advance training was partially effective on the adjusting effect.

• Phonetics and Speech Sciences
• /
• v.15 no.2
• /
• pp.13-20
• /
• 2023

This study explores the potential of automated speech recognition (ASR) in assessing English learners' pronunciation. We employed ASR technology, acknowledged for its impartiality and consistent results, to analyze speech audio files, including synthesized speech, both native-like English and Korean-accented English, and speech recordings from a native English speaker. Through this analysis, we establish baseline values for the word error rate (WER). These were then compared with those obtained for human raters in perception experiments that assessed the speech productions of 30 first-year college students before and after taking a pronunciation course. Our sub-group analyses revealed positive training effects for Whisper, an ASR tool, and human raters, and identified distinct human rater strategies in different assessment aspects, such as proficiency, intelligibility, accuracy, and comprehensibility, that were not observed in ASR. Despite such challenges as recognizing accented speech traits, our findings suggest that digital tools such as ASR can streamline the pronunciation assessment process. With ongoing advancements in ASR technology, its potential as not only an assessment aid but also a self-directed learning tool for pronunciation feedback merits further exploration.

• Journal of The Korean Society of Clinical Toxicology
• /
• v.15 no.1
• /
• pp.31-39
• /
• 2017

Purpose: Despite the clinical and socio-economic impact of acute poisoned patients, many of the treatments are not standardized in Korea. Moreover, no formal training that is specifically focused on clinical toxicology exists. Rather, training and education are conducted case by case in various institutions. This study was conducted to develop a standardized simulation-based clinical toxicology training curriculum for healthcare providers. This program will focus on specific assessment and treatment of critical toxicology patients, specifically those who have been poisoned with organophosphate. Methods: The study was performed using a pre- and post-design to determine the effects of implementation of this program. The study was conducted at eight different urban teaching hospitals in a simulated room in the clinical area. The study was targeted to 19 groups composed of emergency residents and nurses. Simulation-based learning was conducted for each group. Results: All 19 groups achieved the minimum passing score of 75%. Implementation of the program led to improved performance rates for overall management and cooperative moods competency (p<0.01). Inter-rater agreement between the two evaluators was excellent. In general, the participants thought the program was realistic and were able to recognize and improve the competencies needed to care for organophosphate poisoned patients. Conclusion: Simulation-based learning is an effective educational strategy that can be applied to improving and understanding proper care for rare but critical patients. This program was effective at improving team performance and cooperative moods when managing an organophosphate poisoned patient in the Emergency Department.

• Korean Journal of Radiology
• /
• v.25 no.1
• /
• pp.11-23
• /
• 2024

Objective: To investigate whether reader training improves the performance and agreement of radiologists in interpreting unenhanced breast magnetic resonance imaging (MRI) scans using diffusion-weighted imaging (DWI). Materials and Methods: A study of 96 breasts (35 cancers, 24 benign, and 37 negative) in 48 asymptomatic women was performed between June 2019 and October 2020. High-resolution DWI with b-values of 0, 800, and 1200 sec/mm² was performed using a 3.0-T system. Sixteen breast radiologists independently reviewed the DWI, apparent diffusion coefficient maps, and T1-weighted MRI scans and recorded the Breast Imaging Reporting and Data System (BI-RADS) category for each breast. After a 2-h training session and a 5-month washout period, they re-evaluated the BI-RADS categories. A BI-RADS category of 4 (lesions with at least two suspicious criteria) or 5 (more than two suspicious criteria) was considered positive. The per-breast diagnostic performance of each reader was compared between the first and second reviews. Inter-reader agreement was evaluated using a multi-rater κ analysis and intraclass correlation coefficient (ICC). Results: Before training, the mean sensitivity, specificity, and accuracy of the 16 readers were 70.7% (95% confidence interval [CI]: 59.4-79.9), 90.8% (95% CI: 85.6-94.2), and 83.5% (95% CI: 78.6-87.4), respectively. After training, significant improvements in specificity (95.2%; 95% CI: 90.8-97.5; P = 0.001) and accuracy (85.9%; 95% CI: 80.9-89.8; P = 0.01) were observed, but no difference in sensitivity (69.8%; 95% CI: 58.1-79.4; P = 0.58) was observed. Regarding inter-reader agreement, the κ values were 0.57 (95% CI: 0.52-0.63) before training and 0.68 (95% CI: 0.62-0.74) after training, with a difference of 0.11 (95% CI: 0.02-0.18; P = 0.01). The ICC was 0.73 (95% CI: 0.69-0.74) before training and 0.79 (95% CI: 0.76-0.80) after training (P = 0.002). Conclusion: Brief reader training improved the performance and agreement of interpretations by breast radiologists using unenhanced MRI with DWI.