• Journal of The Korean Association For Science Education
• /
• v.22 no.1
• /
• pp.176-189
• /
• 2002

The purpose of the study is to develop major types of scoring rubrics of portfolio system, and estimate the reliability of the rubrics developed. The portfolio system was developed by Science Education Laboratory, Chongju National University of Education in summer, 2000. The portfolio is based on the Unit 2, The Layer and Fossil, and Unit 4, Heat and Change of Objects at fourth-grade level. Four types of scoring rubrics, holistic-general, holistic-specific, analytical-general, and analytical-specific, were developed. Students' portfolios were scored and inter-rater and intra-rater reliability were calculated. To estimate inter-rater reliability, 3 elementary teachers per each rubric(total 12) scored 12 students' portfolios. Teachers who used analytical-specific rubric scored only six portfolios because it took much more time than other rubrics. To estimate intra-rater reliability, second scoring was administered by two raters per rubric in two and half month. The results show that holistic-general rubric has high inter-rater and moderate intra-rater reliability. Holistic-specific rubric shows moderate inter- and intra-rater reliability. Analytical-general rubric has high inter-rater and moderate intra-rater reliability. Analytical-specific rubric shows high inter- and intra-rater reliability. The raters feel that general rubrics seems to be practical but not clear. Specific rubrics provide more clear guidelines for scoring but require more time and effort to develop the rubrics. Analytical-specific rubric requires more than two times of time to score each portfolio and is proved to be highly reliable but less practical.

• Journal of the Korean earth science society
• /
• v.23 no.8
• /
• pp.632-639
• /
• 2002

The purpose of the study is to develop a scoring rubric for students’ mind maps. The participants of this research were students in two fourth-grade classes selected from an elementary school in Pyungtaek-shi. After receiving basic training, students developed mind maps four times while teaming two science units. In order to score the mind maps, a scoring rubric was developed. To estimate the reliability of the rubric, selected mind maps were marked by three teachers and correlational coefficients were calculated with SPSS. As a result of the study, a scoring rubric consisted of three domains, central circle, branches, and expression were developed. The reliability of the rubric is proven to be high to very high.

• Annual Conference on Human and Language Technology
• /
• 2012.10a
• /
• pp.138-141
• /
• 2012

한국어 서답형 문항의 자동채점 프로그램을 개발하기 위해서는 모범답안, 오답, 부분점수 부여를 위한 세부적인 내용을 채점 기준표로 기술해야 한다. 자동채점에 필요한 구체적인 사항들을 기술하기 위하여 XML 형식으로 정답 템플릿을 정의하였다. 채점에 필요한 내용을 단위 개념으로 정의하고 이를 컴퓨터가 엑세스 가능한 형태의 정답 템플릿을 설계하였다. 정답 템플릿 형식에 맞게 편리하게 템플릿을 작성할 수 있는 작성 도구를 이용하여 학업 성취도평가 각 문항에 대한 채점 기준표를 정답 템플릿으로 작성하여 채점기준표를 작성하는 실험을 수행하였다.

• Science of Emotion and Sensibility
• /
• v.12 no.4
• /
• pp.557-564
• /
• 2009

For scoring polygraph charts, the Prosecutors' Office of the Republic of Korea uses a consistency criterion in which an elevated signal on one physiological channel is scored as a deceptive response only if the signal is also elevated on other channels. In the current study, the effects of this scoring criterion on reliability and accuracy (validity) of polygraph scores were assessed. Polygraph tests on 26 suspects were evaluated twice by the same examiners. The examiners used the consistency criterion in the first evaluation. In the second evaluation, the examiners were prevented from using the criterion; the signals from each physiological channel were separated and randomly arranged before they were rescored by the same examiner. Reliability was assessed by the variation among the scores for each suspect. Accuracy was assessed by establishing a standard, based on a Latent Class Analysis model, using the results of polygraph tests on each of 182 additional suspects. Reliability and accuracy were both improved by the use of the consistency criterion which therefore was recommended.

• KIISE Transactions on Computing Practices
• /
• v.22 no.5
• /
• pp.218-224
• /
• 2016

This paper proposes a computer-based practical automatic scoring system for Korean short answers through student answer analysis and natural language processing techniques. The proposed system reduces the overall scoring time and budget, while improving the ease-of-use to write answer templates from student answers as well as the accuracy and reliability of automatic scoring system. To evaluate the application of the automatic scoring system and compare to the human scoring process, we performed an experiment using the student answers of social science subject in 2014 National Assessment of Educational Achievement.

• Proceedings of the Korean Operations and Management Science Society Conference
• /
• 2003.05a
• /
• pp.580-585
• /
• 2003

정보화 수준을 검정하는 다양한 영역 중 실기시험은 채점 강의 문제로 많은 비용과 시간이 소요되고 있다. 본 연구에서는 컴퓨터활용능력을 검정하는 Excel 실기시험을 자동으로 채점하는 웹 기반 채점엔진을 개발하고자 한다. 출제자의 출제의도를 반영한 채점항목 및 채점기준을 유연하게 등록할 수 있게 하고, 틀리거나 감점된 문항에 대해 오답의 이유를 정확하게 제공하는 데에 개발의 초점을 맞추었다. 엑셀 자동채점 엔진을 활용하면 Excel 실기시험의 출제방향에 크게 위배되지 않으면서 사람이 채점타는 경우보다 객관성, 정확성 비용 소요시간의 면에서 기대효과를 꾀할 수 있다.

• Journal of The Korean Association For Science Education
• /
• v.42 no.2
• /
• pp.239-252
• /
• 2022

This study aims to develop performance criteria based on characteristics observed in science inquiry tasks for elementary school students. First, the performance characteristics by observing 70 fifth-grade elementary school students' science inquiry activity report are listed. Second, the checklist-type scoring criteria in connection with the theoretical framework of scientific inquiry process and relevant competencies are developed. Third, with the developed scoring criteria, 11 raters participate in scoring 350 students' reports. The main findings are as follow: first, the scoring data are well-fitted for the many-faceted Rasch model, and 22 scoring criteria are reasonably-well differentiated for various levels of proficiency. Second, at low performance level, observable characteristics are to answer questions explicitly required by the task or to observe objects or phenomena using pre-learned scientific concepts, while at high performance level, to explore additional data other than given data or to reflect on one's experimental process. Based on the results, the usefulness of analyzing students' performance characteristics for developing the scoring criteria, and further research directions are discussed.

• Journal of The Korean Association For Science Education
• /
• v.32 no.1
• /
• pp.160-181
• /
• 2012

The purpose of this study is to understand raters' errors in rating performance assessments of science inquiry. For this, 60 middle school students performed scientific inquiry about sound propagation and 4 trained raters rated their activity sheets. Variance components estimation for the result of the generalizability analysis for the person, task, rater design, the variance components for rater, rater by person and rater by task are about 25%. Among 4 raters, 2 raters' severity is higher than the other two raters and their severities were stabilized. Four raters' rating agreed with each other in 51 cases among the 240 cases. Through the raters' conferences, the rater error types for 189 disagreed cases were identified as one of three types; different salience, severity, and overlooking. The error type 1, different salience, showed 38% of the disagreed cases. Salient task and salient assessment components are different among the raters. The error type 2, severity, showed 25% and the error type 3, overlooking showed 31%. The error type 2 seemed to have happened when the students responses were on the borders of two levels. Error type 3 seemed to have happened when raters overlooked some important part of students' responses because she or he immersed her or himself in one's own salience. To reduce the above rater errors, raters' conference in salience of task and assesment components are needed before performing the holistic scoring of complex tasks. Also raters need to recognize her/his severity and efforts to keep one's own severity. Multiple raters are needed to prevent the errors from being overlooked. The further studies in raters' tendencies and sources of different interpretations on the rubric are suggested.

• Annual Conference on Human and Language Technology
• /
• 2012.10a
• /
• pp.69-72
• /
• 2012

본 논문은 한국어 서답형(단어, 구 수준) 문항 유형을 분석하고 실제 채점자가 채점 기준표를 보고 채점하는 방법을 컴퓨터가 인식할 수 있도록 정답 템플릿을 설계 및 개념 정의를 하여 한국어 서답형에 특화된 자동채점 시스템 방법을 제시한다. 본 시스템을 사용하여 1000개의 학생 답안지에 대한 유형 가지수 500개 이하의 2011년도 학업성취도 평가 과학 6개 문항에 대하여 채점 기준표 내용을 정답 템플릿으로 작성한 뒤 250개 학생 답안을 학습데이터로, 정답 템플릿을 업데이트로 사용, 750개 학생 답안에 대하여 자동채점한 결과, 평균 카파계수 0.84라는 수치로서 실제 사람 채점 결과와 거의 완벽히 일치라는 결과를 얻었다.

• The Journal of Korean Association of Computer Education
• /
• v.21 no.6
• /
• pp.83-92
• /
• 2018

The subjective question is appropriate for evaluation of deep thinking, but it is not easy to score. Since, regardless of same scoring criterion, the graders are able to produce different scores, we need the objective automatic evaluation system. However, the system has the problem of Korean analysis and comparison. This paper suggests the Korean syntactic analysis and subjective grading system using the syntactic-semantic tree comparator. This system is the hybrid grading system of word based and syntactic-semantic tree based grading. This system grades the answers on the subjective question using the syntactic-semantic comparator. This proposed system has the good result. This system will be utilized in Korean syntactic-semantic analysis, subjective question grading, and document classification.