• 한국어교육
• /
• 제28권1호
• /
• pp.173-196
• /
• 2017

Writing is a subjective and performative activity. Writing ability has multi-facets and compoundness. To understand the examinees's writing ability accurately and provide effective writing scores, raters first ought to have the competency regarding assessment. Therefore, this study is significant as a fundamental research about rater's characteristics on the TOPIK writing assessment. 150 scripts of the 47th TOPIK examinees were selected randomly, and were further rated independently by 20 raters. The many-facet Rasch model was used to generate individualized feedback reports on each rater's relative severity and consistency with respect to particular categories of the rating scale. This study was analyzed using the FACETS ver 3.71.4 program. Overfit and misfit raters showed many difficulties for noticing the difference between assessment factors and interpreting the criteria. Writing raters appear to have much confusion when interpreting the assessment criteria, and especially, overfit and misfit teachers interpret the criteria arbitrarily. The main reason of overfit and misfit is the confusion about assessment factors and criteria in finding basis for scoring. Therefore, there needs to be more training and research is needed for raters based on this type of writing assessment characteristics. This study is recognized significantly in that it collectively examined writing assessment characteristics of writing raters, and visually confirmed the assessment error aspects of writing assessment.

• 영어어문교육
• /
• 제16권1호
• /
• pp.123-142
• /
• 2009

The present study used multi-faceted Rasch measurement to explore the characteristics and bias patterns of non-native raters when they scored L2 writing tasks. Three raters scored 254 writing tasks written by Korean university students on two topics adapted from the TOEFL Test of Written English (TWE). The written products were assessed using a five-category rating scale (Content, Organization, Language in Use, Grammar, and Mechanics). The raters only showed a difference in severity with regard to rating categories but not in task types. Overall, the raters scored Grammar most harshly and Organization most leniently. The results also indicated several bias patterns of ratings with regard to the rating categories and task types. In rater-task bias interactions, each rater showed recurring bias patterns in their rating between two writing tasks. Analysis of rater-category bias interaction showed that the three raters revealed biased patterns across all the rating categories though they were relatively consistent in their rating. The study has implications for the importance of rater training and task selection in L2 writing assessment.

• 한국과학교육학회지
• /
• 제32권1호
• /
• pp.160-181
• /
• 2012

본 연구의 목적은 과학탐구활동 수행평가 시 총체적 채점의 신뢰도를 높이기 위하여 채점자간 불일치의 정도와 유형을 이해하는 것이다. 이를 위하여 중학생 60명을 대상으로 과학탐구 수행평가를 실시하였고, 4명의 훈련된 채점자 채점을 실시하였다. 분산 분석결과 교사 관련 분산성분에 의해 전체 분산의 25%를 설명할 수 있으며, 4명의 채점자중 2명은 관대한 채점자, 2명은 엄격한 채점자의 경향을 지닌 것으로 나타났다. 전체 240 채점 사례 중 4명의 채점자가 모두 일치한 사례는 51사례이다. 채점자간 불일치가 나타나는 189사례에 대하여 채점자 협의를 통하여 확인한 결과, 채점자간 중요하게 생각하는 부분의 차이 때문에 발생하는 불일치 유형1이 38%, 채점자의 관대함과 엄격함에 의해 발생하는 불일치 유형2가 25%, 채점자가 중요하게 생각하지 않은 부분에서 간과하기 때문에 발생하는 등 실수에 의한 불일치 유형3이 31%로 나타났다. 불일치 유형1은 채점자마다 중요하게 생각하는 과제 요소와 평가 요소가 다른 경우로 나누어서 나타났으며, 맥락상의 의미를 강조하는 채점자는 관대한 경향을, 특정 요소를 강조하여 분석적으로 해석하는 채점자는 엄격한 경향을 나타냈다. 불일치 유형 2는 많은 경우 채점 척도의 경계에 학생의 응답에 대하여 나타났으며, 채점자들은 이러한 학생 응답에 대하여 옳은 서술의 개수를 세는 등 분석적인 채점을 수행하는 것을 확인할 수 있었다. 또한 불일치 유형3은 채점자의 실수로 발생하는 불일치로 주로 학생의 응답 중 평가 기준에 부합하는 부분인데 채점자가 중요하게 생각하지 않기 때문에 간과하여 발생하는 것으로 파악할 수 있었다. 이상과 같은 채점자간 불일치를 제어하기 위해서는 채점자가 중요하게 생각하는 과제 요소와 평가 요소에 대하여 사전 및 진행 중 협의를 할 필요가 있다고 판단된다. 또한 총체적 채점을 하는 경우도 각 수준에 해당하는 평가 기준과 함께 경계에 놓인 학생 응답을 변별하는 기준을 제시하는 것이 필요하다. 채점자들은 자신의 채점경향이 엄격한지 관대한지를 파악하고 경계에 놓인 학생의 응답에 대한 판단을 주의 깊게 하여야 불일치를 줄일 수 있다. 실수에 의한 오차를 줄이기 위해서는 여러 명의 채점자가 교차 채점하는 것이 필요하다. 동일한 채점 기준에 대한 채점자의 해석이 다르게 나타나는 경향과 원인에 대한 추후 연구가 필요하다.

• 중환자간호학회지
• /
• 제5권2호
• /
• pp.61-71
• /
• 2012

Purpose: The aim of this study was to measure the inter-rater reliability of Emergency severity index (ESI) version 4 among triage nurse. Methods: This study was carried out from August 11, 2010 to September 7, 2010 in a regional emergency department. Data collection was done by ten triage nurses who trained ESI v.4. Two research nurses and ten triage nurses scored the ESI version 4 to the patients as references, independently. We calculated the weighted kappa between the triage nurses and research nurses to evaluate the consistency of the ESI v.4. Results: A total of 233 patients were enrolled in this study. Classification of ESI level was as follows - level 1 (0.4%), level 2 (21.0%), level 3 (67.8%), level 4 (9.4%), and level 5 (1.3%). Inter-rater reliability by weighted kappa was 0.79 (95% Confidence Interval= 0.74-0.83) and agreement rate was 87.1%. Under-triage rate by triage nurse was 6.0% and over-triage rate was 6.9%. Conclusion: For this study, inter-rater reliability was measured good level between triage nurses and research nurses in Korean single ED.

• 대한후두음성언어의학회지
• /
• 제18권2호
• /
• pp.96-101
• /
• 2007

Laryngopharyngeal reflux (LPR) is the retrograde movement of gastric contents into the larynx, pharynx, and upper aero-digestive tract. LPR differs from gastroesophageal reflux in that it is often not associated with heartburn and regurgitation symptoms. Otolaryngological manifestations of acid reflux include a wide range of pharyngeal and laryngeal symptoms. Belafsky et al. developed a useful self-administered tool, the reflux symptom index (RSI), for assessing the degree of LPR symptoms. Patients are asked to use a 0 to 5 point scale to grade the following symptoms: 1) hoarseness or voice problems; 2) throat clearing; 3) excess throat mucus or postnasal drip ; 4) difficulty swallowing; 5) coughing after eating or lying down; 6) breathing difficulties ; 7) troublesome or annoying cough; 8) sensation of something sticking or a lump in the throat; 9) heartburn, chest pain, indigestion or stomach acid coming up. A RSI score greater than 13 is considered abnormal. As there is no validated instrument to document the physical findings and severity of LPR, Belafsky et al. developed an eight-item clinical severity scale for judging laryngoscopic finding, the reflux finding score (RFS). They rated eight LPR-associated findings on a scale from 0 to 4 : subglottic edema, ventricular obliteration, erythema/hyperemia, vocal-fold edema, diffuse laryngeal edema, posterior commissure hypertrophy, granuloma/granulation tissue, and thick endolaryngeal mucus. A RFS score of greater than 7 was found to suggest LPR-associated laryngitis. Although both indices (RSI and RFS) are widely used, there is some controversy about their validity (sensitivity and specificity) and reliability (intra-rater and inter-rater) in LPR diagnosis and treatment. We discuss the validity and reliability of RSI and RFS with literature review.

• 말소리와 음성과학
• /
• 제1권4호
• /
• pp.165-170
• /
• 2009

This study was to investigate for comparison between single rating and average ratings from multiple presentations of the same stimulus for measuring the voice quality of dysphonia using 7-point equal-appearing interval (EAI) rating scale. Overall severity of voice quality for 46 /a/ vowel stimuli (23 stimuli from dysphonia, 23 stimuli from control) was rated by 3 experienced speech-language pathologists (averaged 19 years; range = 7 to 40 years). For average ratings, each stimulus was rated five times in random order and averaged from two to five times. Although higher inter-rater reliability was found in average ratings than in single rating, there were no significant differences in rating scores between single and multiple average ratings judged by experienced listeners, suggesting that auditory perceptual ratings judged by well-trained listeners have relatively good agreement with the same stimulus across the judgment. Larger variations in perceptual ratings were observed for moderate voices than for mild or severe voices, even in the average ratings.

• 동의신경정신과학회지
• /
• 제31권1호
• /
• pp.1-12
• /
• 2020

Objectives: The purpose of this study was to evaluate the reliability and validity of the instrument on pattern identification for insomnia (PIT-Insomnia) and verify the correlation between PIT-Insomnia and psychological tests. Methods: Two evaluators examined the pattern identification of the participants who met insomnia disorder diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorder, Fifth Edition (DSM-5) and took the Insomnia Severity Index (ISI) score over 15 once manually and twice using the PIT-Insomnia to measure the inter-rater and test-retest reliability. We also conducted the following surveys: the Pittsburgh Sleep Quality Index (PSQI), the Korean version of Beck's depression inventory (K-BDI), the Korean version of the State-Trait Anxiety Inventory (STAI-K), the Korean Symptom checklist-95 (KSCL-95), and the EuroQol-5 dimension (EQ-5D), to measure concurrent validity and correlation between the PTI-Insomnia and psychological tests. Results: 1. The test-retest reliability analysis of the pattern identification results showed moderate agreement, and test-retest reliability analysis of each pattern identification score showed agreements from poor to moderate. 2. The inter-rater reliability analysis of the pattern identification results via manual showed slight agreement, when analysis was performed with calibration, the inter-rater reliability analysis of the pattern identification results via manual showed fair agreement. 3. The concordance analysis between results via manual and the PIT-Insomnia showed poor agreement, when the analysis was performed with calibration, concordance analysis showed fair agreement. 4. The concordance analysis between the PIT-Insomnia and the PSQI showed positive linear correlation. 5. The concordance analysis between the PIT-Insomnia and the PSQI, K-BDI, STAI-K, KSCL-95, and EQ-5D showed that non-interaction between the heart and kidney have positive linear correlation with the K-BDI, anxiety item of KSCL-95, dual deficiency of the heart-spleen have positive linear correlation with somatization item of KSCL-95, paranoia item of KSCL-95, heart deficiency with timidity have positive linear correlation with stress vulnerability item of KSCL-95, parania item of KSCL-95, phlegm-fire harassing the heart have positive linear correlation with K-BDI, paranoia item of KSCL-95, depressed liver qi transforming into fire have positive linear correlation with the anxiety item of KSCL-95, parania item of KSCL-95, all pattern identification have negative linear correlation with EQ-5D. Conclusions: The PIT-Insomnia has moderate agreement of reliability and reflects the severity of insomnia since it has some concurrent validity with the PSQI. There are some correlations between the PTI-Insomnia with specific psychological tests, so we could suggest it can be used appropriately in the clinical situation.