Journal of Digital Policy (디지털정책학회지)

The Society of Digital Policy and Management (한국디지털정책학회)
권호 표지
DOI QR코드

DOI QR Code

An Exploratory Study on ChatGPT's Performance to Answer to Police-related Traffic Laws: Using the Driver's License Test and the Road Traffic Accident Appraiser

ChatGPT의 경찰 관련 교통법규 응답 능력에 대한 탐색적 연구 - 운전면허 학과시험과 도로교통사고감정사 1차 시험을 대상으로 -

  • Sang-yub Lee (Department of Police, Korean National Police University)
  • Received : 2023.10.27
  • Accepted : 2023.12.28
  • Published : 2023.12.28
Download PDF
⟨ Previous Next ⟩

Abstract

This study conducted preliminary study to identify effective ways to use ChatGPT in traffic policing by analyzing ChatGPT's responses to the driver's license test and the road traffic accident appraiser test. I collected ChatGPT responses for the driver's license test item pool and the road traffic accident appraiser test using the OpenAI API with Python code for 30 iterative experiments, and analyzed the percentage of correct answers by test, year, section, and consistency. First, the average correct answer rate for the driver's license test and the for road traffic accident appraisers test was 44.60% and 35.45%, respectively, which was lower than the pass criteria, and the correct answer rate after 2022 was lower than the average correct answer rate. Second, the percentage of correct answers by section ranged from 29.69% to 56.80%, showing a significant difference. Third, it consistently produced the same response more than 95% of the time when the answer was correct. To effectively utilize ChatGPT, it is necessary to have user expertise, evaluation data and analysis methods, design a quality traffic law corpus and periodic learning.

본 연구는 경찰교통에서의 효과적 ChatGPT 활용 방안 도출을 위한 사전 연구로서 운전면허 학과시험과 도로교통사고감정사 시험에 대한 ChatGPT의 응답을 분석하였다. ChatGPT가 뛰어난 성능과 접근성으로 여러 분야에서 기대를 받고 있으나 경찰 교통법규와 같이 고도의 정확성이 요구되는 분야에서는 사전에 그 성능과 한계를 탐색할 필요가 있다. 이에 본 연구에서는 운전면허 학과시험 문제은행과 도로교통사고감정사 1차 시험을 대상으로 파이썬 코드로 OpenAI API를 이용해 30회의 반복 실험으로 ChatGPT의 응답을 수집하고 응답 결과를 바탕으로 시험별·연도별·내용 영역별 정답률, 일관성 능력을 분석하였다. 분석 결과 첫째, 운전면허 학과시험 및 도로교통사고감정사 1차 시험의 평균 정답률은 각 44.60%, 35.45%로 합격기준보다 낮았다. 연도별로는 2022년 이후 정답률이 평균 정답률을 하회했다. 둘째, 영역별 정답률은 29.69%~56.80%로 나타나 큰 편차를 보였다. 셋째, 정답을 맞힌 경우 95% 이상 일관되게 같은 응답을 출력하였다. ChatGPT의 효과적 활용을 위해서는 사용자의 전문 지식, 평가 데이터 및 방법 마련, 양질의 교통법규 말뭉치 설계와 주기적 학습이 필요하다고 판단된다.

Keywords

References

  1. Science & Technology Policy Institute(2023), The Generative AI Era Triggered by ChatGPT. What the Future Holds and How to Respond, 02-09. 
  2. O. N. Kwon, S. J. Oh, J. Yoon, K. Lee & B. C. Shin. (2023), "Analyzing Mathematical Performances of ChatGPT: Focusing on the Solution of National Assessment of Educational Achievement and the College Scholastic Ability Test. Communications of Mathematical Education, 37(2), 233-256. DOI : 10.7468/jksmee.2023.37.2.233 
  3. Y. Lee, C. Kim & H. Ahn. (2023). A Study on the ChatGPT: Focused on the News Big Data Service and ChatGPT Use Cases. Journal of the Korea Society of Digital Industry and Information Manageme, 19(1), 139-151. DOI : 10.17662/ksdim.2023.19.1.139 
  4. D. H. Kim (2023. 1. 28). The Chosunilbo. https://www.chosun.com/politics/politics_general/2023/01/28/3DVO3AS4CVGDJMAOUCO3N3PAOI/?utm_source=naver&utm_medium=referral&utm_campaign=naver-news. 
  5. G. Kim, J. Cho, S. Kim, S. Beak, S. Ryu, J. Koh & B. Kim. (2021). Deep Learning-based Real-time Traffic Accident Type and Fault Information Provision Service. The Journal of The Institute of Internet, Broadcasting and Communication, 21(3), 1-6 DOI : 10.7236/JIIBC.2021.21.3.1 
  6. S. J. Lim & Y. S. Shin. (2020). A Study on Traffic Violation Surveillance System Based on Edge AI -Detecting a car which cuts in left-turn waiting lines. KICS Summer Conference 2020(pp.1315-1316) 
  7. G. S. Lee. (2020). Illegal Parking Number Recognition Technology using Deep Learning Algorithm Based on Drone Image. Journal of The Korean Cadastre Information Association, 22(3), 20-31 DOI : 10.46416/JKCIA.2020.12.22.3.20 
  8. Y. J. Roh & S. H. Bae. (2021), "Forecasting of Traffic Accident Occurrence Pattern Using LSTM", The Journal of The Korea Institute of Intelligent Transportation Systems, 20(3), 59-73 DOI : 10.12815/kits.2021.20.3.59 
  9. J. D. Rye, S. Park, S. Park, C. Kwon & I. Yun. (2018). A Study for Development of Expressway Traffic Accident Prediction Model Using Deep Learning. The Journal of The Korea Institute of Intelligent Transportation Systems, 17(4), 14-25 DOI : 10.12815/kits.2018.17.4.14 
  10. S. Huh. (2023). Can we trust AI chatbots' answers about disease diagnosis and patient care?. Journal of the Korean Medical Association, 66(4), 218-222 DOI : 10.5124/jkma.2023.66.4.218 
  11. S. K. Kwon & Y. T. Lee. (2023). Investigating the performance of generative AI ChatGPT's reading comprehension ability. Journal of the Korea English Education Society, 22(2), 147-172 DOI : 10.18649/jkees.2023.22.2.147 
  12. L. Ouyang et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35(pp.27730-27744). DOI : 10.48550/arXiv.2203.02155 
  13. OpenAI. (n.d). https://platform.openai.com/docs/models/gpt-3-5. 
  14. M. Bommarito II & D. M. Katz. (2022). GPT takes the bar exam. arXiv preprint arXiv:2212.14402. DOI : 10.48550/arXiv.2212.14402 
  15. I. Chalkidis. (2023). ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark. arXiv preprint arXiv:2304.12202. DOI : 10.48550/arXiv.2304.12202 
  16. J. H. Choi, K. E. Hickman, A. Monahan, & D. Schwarcz. (2023). Chatgpt goes to law school. Available at SSRN. DOI : 10.2139/ssrn.4335905 
  17. S. Park, J. Park & J. Ahn. (2023). Potential Applications and Implications of GPT-4 in Legal Inference Using Korean Legal Aptitude Test (LEET). Journal of Law & Economic Regulation, 16(1), 7-28 DOI : 10.22732/CeLPU.2023.16.1.7 
  18. M. Baek. (2023). A Study on the Assessment of Korean Language Proficiency of ChatGPT - Focusing on the Reading Section of TOPIK and Idioms -. Language Facts and Perspectives, 59, 279-308 DOI : 10.20988/lfp.2023.59..279 
  19. J. J. Yu. (2023). Application of Artificial Intelligence for Geography Education - Focusing on Question Answering on ChatGPT -. Journal of the Association of Korean Photo-Geographers, 33(1), 162-173 DOI : 10.35149/jakpg.2023.33.1.011 
  20. Y. Yoon. (2023). Artificial Intelligence and Accounting Education:Focusing on ChatGPT and Its Applications. Korean Computers and Accounting Review, 21(1), 1-29 DOI : 10.32956/kaoca.2023.21.1.1 
  21. J. Kim. (2013). A study on the Development Plan of Driver's License System - Focusing on the Test for Driver's License -. Master's Thesis, Graduate School of Transportation / Intelligent Transport Systems 
  22. S. Y. Baik. (2018). A Study on the Driver's License Written Test System to Improve New Driver' Driving Ability. The Korean Association of Police Science Review, 20(2), 31-66  https://doi.org/10.24055/kaps.20.2.2
  23. Korean National Police Agency. (n.d). Korean National Police Agency 2021 White Paper, 236 
  24. Statistics Korea. (n.d). https://kosis.kr/statHtml/statHtml.do?orgId=132&tblId=DT_13201_A002&conn_path=I3. 
  25. KoROAD. (n.d). https://www.safedriving.or.kr/license/licPass.do?menuCode=MN-PO-1541. 
  26. KRIVET. (n.d). https://www.pqi.or.kr/inf/qul/infQulBasDetail.do?qulId=811. 
  27. Santiago. (n.d). https://twitter.com/svpino/status/1610984481342771200.