Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of an Automatic Scoring Model Using a Voting Method for Descriptive Answers

투표 기반 서술형 주관식 답안 자동 채점 모델의 설계 및 구현

  • Received : 2013.06.03
  • Accepted : 2013.08.02
  • Published : 2013.08.30
Download PDF
⟨ Previous Next ⟩

Abstract

TIn this paper, we propose a model automatically scoring a student's answer for a descriptive problem by using a voting method. Considering the model construction cost, the proposed model does not separately construct the automatic scoring model per problem type. In order to utilize features useful for automatically scoring the descriptive answers, the proposed model extracts feature values from the results, generated by comparing the student's answer with the answer sheet. For the purpose of improving the precision of the scoring result, the proposed model collects the scoring results classified by a few machine learning based classifiers, and unanimously selects the scoring result as the final result. Experimental results show that the single machine learning based classifier C4.5 takes 83.00% on precision while the proposed model improve the precision up to 90.57% by using three machine learning based classifiers C4.5, ME, and SVM.

본 논문에서는 투표기법을 이용하여 서술형 주관식 문제에 대한 학습자 답안을 자동으로 채점하는 모델을 제안한다. 제안하는 방법은 모델 구축 비용을 줄이기 위해서, 문제 유형별로 세분화하여 서술형 주관식 답안 자동 채점 모델을 따로 구축하지 않는다. 제안하는 방법은 서술형 주관식 답안 자동 채점에 유용한 자질을 추출하기 위해서, 모범 답안과 학습자 답안을 비교한 결과를 바탕으로 다양한 자질을 추출한다. 제안하는 방법은 답안 채점 결과의 신뢰성을 높이기 위해서, 각 학습자 답안을 여러 기계학습 기반 분류기를 이용하여 채점하고, 각 채점 결과를 투표하여 만장일치로 선택한 채점 결과를 최종 채점 결과로 결정한다. 실험결과 기계학습 기반 분류기 C4.5만 사용한 채점 결과는 정확률이 83.00%인데 반해, 기계학습 기반 분류기 C4.5, ME, SVM에서 만장일치로 선택한 채점 결과는 정확률이 90.57%까지 개선되었다.

Keywords

References

  1. C. G. Jung, R. I. Choi, "A Developmental Plan on Public Education in Digital Times, 21c," Journal of The Korea Society of Computer and Information, Vol. 8, No. 1, pp.120-128, Mar. 2003.
  2. H. J. Kim, J. H. Choi, "Development of a Teaching and Learning Model for Educational Usage of Web 2.0 and Its Effect Analysis," Journal of The Korea Society of Computer and Information, Vol. 16, No. 10, pp.45-52, Oct. 2011. https://doi.org/10.9708/jksci.2011.16.10.045
  3. K. A. Jin, "Development of automated scoring system for English writing." English Language & Literature Teaching, Vol. 13, No. 1, pp.235-259, Spring. 2007.
  4. K. B. Kim, J. H. Cho, "Performance Assessment System using Fuzzy Reasoning Rule," Journal of The Korea Society of Computer and Information, Vol. 10, No. 1, pp.209-216, Mar. 2005.
  5. O. Y. Kwon, "The Design and Implementation of Web-based Subjective questions Grading Algorithm," Department of Educational(Coumputer Scientce) The Graduate School of Education. Hanseo University, pp.1-3(52), Feb. 2004.
  6. W. S. Kang, "Design and Implementation of a Subjective-type Evaluation System Using Syntactic and Case-Role Information," THE JOURNAL OF KOREAN ASSOCIATION OF COMPUTER EDUCATION, Vol. 10, No. 5, pp.61-69, Sept. 2007.
  7. H. J. Park, W. S. Kang, "Design and Implementation of a Subjective-type Evaluation System Using Natural Language Processing Technique," THE JOURNAL OF KOREAN ASSOCIATION OF COMPUTER EDUCATION, Vol. 6, NO. 3, pp.207-216, Jul. 2003.
  8. E. H. No, "Developing an Automatic Content Scoring Program for Short Answer Korean Items in Large-Scalse Assessments," Korea Institute for Curriculum and Evaluation, pp.3-48(230), 2012.
  9. Y. R. Kim, B. Y. Cho, "(A)review on NAEA Korean test based on CURRV framework," Korea Institute for Curriculum and Evaluation, pp.1-12p, 2011.
  10. C. Park, "Validating an Automated Scoring System of Constructed-response items for applications," Journal of Educational Evaluation. Vol. 22, No. 3, pp615-631, Sept. 2009.
  11. E. M. Jung, "Design and Implementation of Automatic Marking System for a Subjectivity Problem of the Program," Journal of Korea Multimedia Society, Vol. 12, No. 5, pp767-776, May. 2009.
  12. J. M. Cho, K. H. Kim, "A Study on design of The Internet-based scoring system for constructed responses," THE JOURNAL OF KOREAN ASSOCIATION OF COMPUTER EDUCATION, Vol. 10, No. 2, pp.89-100, Mar. 2007.
  13. W. J. Cho, "An Intelligent Marking System based on Semantic Kernel and Korean WordNet," Graduate School. Hallym University, pp. 3-33, Aug. 2006.
  14. J. S. Oh, W. J. Cho, Y. S. Kim, J. Y. Lee, "A Descriptive Question Marking System based on Semantic Kernels," Journal of Korean Institute of Information Technology, Vol. 3, No. 4, pp.95-104, Sept. 2005.
  15. D. K. Chung, "Subjective Questions Scoring System Using Vector Similarity and Thesaurus," The Graduate School of Education. Dongguk University, pp.1-45, 2002.
  16. J. H. Cho, H. K. Jung, C. Y. Park, Y. S. Kim, "An Autonomous Assessment of a Short Essay Answer by Using the BLEU," HCI2009, pp.606-605, Gangwon-do Phoenixpark Convention center, Korea, Feb. 2009.
  17. S. H. Jang, "Planning and realizing the Automatic Grading System for Subjective Questions Through an Analysis of Question Types," The Graduate School of Education. Andong National University, 53p, 2008.
  18. Van Halteren, H., Daelemans, W. and Zavrel, J., "Improving accuracy in word class tagging through the combination of machine learning systems," Computational linguistics, Vol. 27, No. 2, pp.199-229, Feb. 2001. https://doi.org/10.1162/089120101750300508
  19. Van Halteren, H., Zavrel, J. and Daelemans, W., "Improving data driven wordclass tagging by system combination," Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Vol. 1, pp.491-497, 1998.
  20. J. T. Lee, Y. I. Song, S. Y. Park, H. C. Rim, "Text-Confidence Feature Based Quality Evaluation Model for Knowledge Q&A Documents," Journal of KIISE : Software and Applications, Vol. 35, No. 10, pp.608-615, Nov. 2008.
  21. J. T. Lee, Y. I. Song, H. C. Rim, "Predicting the quality of answers using surface linguistic features," In Proceedings of the 6th International Conference on Advanced Language Processing and Web Information Technology, pp.111-116, 2007.
  22. Quinlan, J. Ross, "C4.5:Programs for Machine Learning," Morgan Kaufmann Publishers, 1993
  23. H. J. Jung, "A Maximum Entropy Approach to Korean Part-of-Speech Tagging," Information & Communications of Engineering, Myongji University, pp.11-18, 2005.
  24. Vapnik, "The Nature of Statistical Learning Theory," Springer-Verlag, 1999
  25. S. Y. Kim, "Incremental Supervised Learning based on SVM with Unlabeled Documents," The Graduate School of Computer Science. Yonsei University, pp.12-13, 2002.
  26. T. Joachims, "Estimating the Generalization Performance of an SVMEfficiently," Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 431-438, 2000.
  27. S. Z. Lee, "New Statistical Models for Automatic Part-of-Speech Tagging," Ph.D. Thesis, Korea University, 1999.
  28. S. Y. Kim, "Object of Interest Extraction Using Gabor Filters," THE JOURNAL OF KOREAN ASSOCIATION OF COMPUTER EDUCATION, Vol. 13, No. 2, pp.87-94, 2008 https://doi.org/10.1007/978-3-540-70600-7_7

Cited by

  1. 구문의미트리 비교기를 이용한 주관식 문항 채점 시스템에 대한 연구 vol.21, pp.6, 2013, https://doi.org/10.32431/kace.2018.21.6.007