• The Journal of Korean Association of Computer Education
• /
• v.17 no.2
• /
• pp.39-51
• /
• 2014

In high-stakes tests for large groups, the efficiency with which students' responses are distributed to raters and how systematic scoring procedures are managed is important to the overall success of the testing program. In the scoring of constructed response items, it is important to understand whether the raters themselves are making consistent judgments on the responses, and whether these judgments are similar across all raters in order to establish measures of rater reliability. The purpose of this study was to design, develop and carry out a pilot test of an online scoring system for constructed response items administered in a paper-and-pencil test to large groups, and to verify the system's reliability. In this study, we show that this online system provided information on the scoring process of individual raters, including intra-rater and inter-rater consistency, compared to conventional scoring methods. We found this system to be especially effective for obtaining reliable and valid scores for constructed response items.

• Journal of The Korean Association For Science Education
• /
• v.32 no.1
• /
• pp.160-181
• /
• 2012

The purpose of this study is to understand raters' errors in rating performance assessments of science inquiry. For this, 60 middle school students performed scientific inquiry about sound propagation and 4 trained raters rated their activity sheets. Variance components estimation for the result of the generalizability analysis for the person, task, rater design, the variance components for rater, rater by person and rater by task are about 25%. Among 4 raters, 2 raters' severity is higher than the other two raters and their severities were stabilized. Four raters' rating agreed with each other in 51 cases among the 240 cases. Through the raters' conferences, the rater error types for 189 disagreed cases were identified as one of three types; different salience, severity, and overlooking. The error type 1, different salience, showed 38% of the disagreed cases. Salient task and salient assessment components are different among the raters. The error type 2, severity, showed 25% and the error type 3, overlooking showed 31%. The error type 2 seemed to have happened when the students responses were on the borders of two levels. Error type 3 seemed to have happened when raters overlooked some important part of students' responses because she or he immersed her or himself in one's own salience. To reduce the above rater errors, raters' conference in salience of task and assesment components are needed before performing the holistic scoring of complex tasks. Also raters need to recognize her/his severity and efforts to keep one's own severity. Multiple raters are needed to prevent the errors from being overlooked. The further studies in raters' tendencies and sources of different interpretations on the rubric are suggested.

• Proceedings of the Korean Information Science Society Conference
• /
• 2001.04b
• /
• pp.673-675
• /
• 2001

본 논문에서는 근거리통신, 인터넷 등의 통신을 이용하여 해당 출제자의 분야 별로 출제한 주관식문제를 응시하면 학습자의 답안과 채점정보를 출제자에게 의뢰하여 채점결과와 해설을 학습자에게 알려주는 주간식문제채점 시스템에 관한 것으로, 학습자가 입력한 주관식 문제의 답을 채점하기 위해서 출제자에게 전자우편이나 전화로 연락하면 출제자가 이 주관식문제채점 시스템에 접속하여 출제자컴퓨터에 표시된 문제, 학습자의 답, 정답, 유사한 답, 배점, 해설 등을 보고 채점한 점수를 출제자화면에 입력하고, 상기 채점한 점수를 합한 점수와 틀린 문제의 해설을 해당 학습자에게 보여주는 주관식문제 채점보조 시스템에 관한 것이다.

• The Journal of Korean Association of Computer Education
• /
• v.10 no.2
• /
• pp.89-100
• /
• 2007

Scoring the constructed responses in large-scale assessments needs great efforts and time to reduce the various types of error in Paper-based training and scoring. For the purpose of eliminating the complexities and problems in Paper and pencil based training and scoring, many of countries including U.S.A and England already have applied online scoring system. There, however, has been few studies to develop the scoring system for the constructed responses items in Korea. The purpose of this study is to develop the basic design of the Internet-based scoring system for the constructed responses. This study suggested the algorithms for assigning scorers to constructed responses, employing methods for monitoring reliability, etc. This system can ensure reliable, quick scoring such as monitor scorer consistency through ongoing reliability checks and assess the quality of scorer decision making through frequent various checking procedures.

• Journal of The Korean Association For Science Education
• /
• v.22 no.1
• /
• pp.176-189
• /
• 2002

The purpose of the study is to develop major types of scoring rubrics of portfolio system, and estimate the reliability of the rubrics developed. The portfolio system was developed by Science Education Laboratory, Chongju National University of Education in summer, 2000. The portfolio is based on the Unit 2, The Layer and Fossil, and Unit 4, Heat and Change of Objects at fourth-grade level. Four types of scoring rubrics, holistic-general, holistic-specific, analytical-general, and analytical-specific, were developed. Students' portfolios were scored and inter-rater and intra-rater reliability were calculated. To estimate inter-rater reliability, 3 elementary teachers per each rubric(total 12) scored 12 students' portfolios. Teachers who used analytical-specific rubric scored only six portfolios because it took much more time than other rubrics. To estimate intra-rater reliability, second scoring was administered by two raters per rubric in two and half month. The results show that holistic-general rubric has high inter-rater and moderate intra-rater reliability. Holistic-specific rubric shows moderate inter- and intra-rater reliability. Analytical-general rubric has high inter-rater and moderate intra-rater reliability. Analytical-specific rubric shows high inter- and intra-rater reliability. The raters feel that general rubrics seems to be practical but not clear. Specific rubrics provide more clear guidelines for scoring but require more time and effort to develop the rubrics. Analytical-specific rubric requires more than two times of time to score each portfolio and is proved to be highly reliable but less practical.

• Annual Conference on Human and Language Technology
• /
• 2014.10a
• /
• pp.112-116
• /
• 2014

서답형 문항은 학생들의 종합적인 사고능력을 판단하는데 매우 유용하지만 채점할 때, 시간과 비용이 매우 많이 소요되고 채점자의 공정성을 확보해야 하는 어려움이 있다. 이러한 문제를 개선하기 위해 본 논문에서는 서답형 문항에 대한 자동채점 시스템을 제안한다. 본 논문에서 제안하는 시스템은 크게 언어 처리 단계와 채점 단계로 나뉜다. 첫 번째로 언어 처리 단계에서는 형태소 분석과 같은 한국어 정보처리 시스템을 이용하여 학생들의 답안을 분석한다. 두 번째로 채점 단계를 진행하는데 이 단계는 아래와 같은 순서로 진행된다. 1) 첫 번째 단계에서 분석 결과가 완전히 일치하는 답안들을 하나의 유형으로 간주하여 각 유형에 속한 답안의 빈도수가 높은 순서대로 정렬하여 인간 채점자가 고빈도 학생 답안을 수동으로 채점한다. 2) 현재까지 채점된 결과와 모범답안을 학습말뭉치로 간주하여 자질 추출 및 자질 가중치 학습을 수행한다. 3) 2)의 학습 결과를 토대로 미채점 답안들을 군집화하여 분류한다. 4) 분류된 결과 중에서 신뢰성이 높은 채점 답안에 대해서 인간 채점자가 확인하고 학습말뭉치에 추가한다. 5) 이와 같은 방법으로 미채점 답안이 존재하지 않을 때까지 반복한다. 제안된 시스템을 평가하기 위해서 2013년 학업성취도 평가의 사회(중3) 및 국어(고2) 과목의 서답형 문항을 사용하였다. 각 과목에서 1000개의 학생 답안을 추출하여 채점시간과 정확률을 평가하였다. 채점시간을 전체적으로 약 80% 이상 줄일 수 있었고 채점 정확률은 사회 및 국어 과목에 대해 각각 98.7%와 97.2%로 나타났다. 앞으로 자동 채점 시스템의 성능을 개선하고 인간 채점자의 집중도를 높일 수 있도록 인터페이스를 개선한다면 국가수준의 대단위 평가에 충분히 활용할 수 있을 것으로 생각한다.

• Journal of The Korean Association For Science Education
• /
• v.22 no.2
• /
• pp.398-409
• /
• 2002

The purpose of this study was to investigate the validity, reliability and practicality of a concept map as an assessment tool in the context of biology concept learning. Forty undergraduate students participated in concept mapping, and the maps were scored by preservice science teachers, using one of three different scoring methods, that is, concept map scoring methods developed by Burry-Stock, Novak & Gowin and McClure & Bell. Two scorers were assigned to each scoring method. As far as the validity of the assessment methods was concerned, two of the three methods were found to be very valid, while Burry-Stock's scoring method was shown little valid. As far as the internal consistency of the methods was concerned, considerably high consistencies were shown between every pair of scorers, judging from high correlation coefficients between the pair of scorers for each scoring method. It took from 1.13 minutes to 3.70 minutes to assess a map at the average. It showed that concept mapping could be used in school classrooms with the limited resources of time and people. These findings suggest that the concept mapping can be an appropriate tool for assessing biology concept understandings.

• Annual Conference of KIPS
• /
• 2003.11a
• /
• pp.253-256
• /
• 2003

기존의 주관식 문제 채점 보조 시스템에서의 프로그래밍 주관식 문제는 학습자가 프로그래밍을 완성한 후 이를 웹 서버에게 전송하면 웹 서버는 이를 출제자에게 통지하고, 통지 받은 출제자는 학습자의 답안을 직접보고 채점하는 복잡한 채점 보조 시스템이었다. 본 논문에서는 프로그래밍 주관식 문제의 경우 이러한 복잡한 채점 보조 시스템을 탈피하고, 학습자의 프로그래밍 소스를 객관적으로 평가하며, 자동으로 답안을 채점하는 채점 보조 시스템을 제안한다.

• The Journal of the Korea Contents Association
• /
• v.18 no.8
• /
• pp.345-352
• /
• 2018

The purpose of this study is to clarify the difference between the scoring results by scorer(doctors and standardization patients) and examination dates. A total of 101 students in the fourth grade of medical school participated in four clinical practice test. Students were randomly assigned to either day-1 or day-2, which was consisted of a standardized patient scoring set or a physician scoring set. Station checklists consisted of history taking, physical examination, patient education, physician-patient relationship and clinical courtesy. The achievement scores of each case and each domain were converted to the standard score, and the differences between groups were compared. Female students' achievement scores were significantly higher than male students' achievement scores in all domains. There was no significant difference between means by the standardized patients' group and doctors group. Day-2 group was significantly higher than day-1 group in both of history taking and physical examination domains. If the principles of checklist are clearly defined, the scorer status (either physician or standardized patients) does not determine the difference of students' practice test scores.

• Annual Conference of KIPS
• /
• 2002.11b
• /
• pp.1463-1466
• /
• 2002

본 논문은 인터넷과 이동전화 사용자가 늘어감으로써 무선인터넷의 필요성이 대두고 있는 가운데 WAP 환경에서 주관식 채점시스템을 접목시킨 시스템을 구현하고자 한다. 근거리통신, 무선인터넷 등의 통신을 이용하여 학습자가 주제별 주관식문제에 응시한 후 해당 답안 정보는 출제자 무선단말기에 푸쉬(Push) 메시지 형태로 즉시 보내지고, 출제자는 학습자의 답안정보를 받아 무선단말기를 이용하여 학습자가 작성한 답안의 배점을 결정하고 채점결과를 서버에 보낸 후 서버는 이 배점으로 채점결과를 학습자에게 신속하게 알려주는 시스템으로 학습자는 신속한 채점결과를 확인하여 학습능력을 향상시킬 수 있는데 목적이 있다.