Journal of KIBIM (한국BIM학회 논문집)

Korean Institute of Building Information Modeling (한국BIM학회)
권호 표지
DOI QR코드

DOI QR Code

Evaluating ChatGPT's Competency in BIM Related Knowledge via the Korean BIM Expertise Exam

BIM 운용 전문가 시험을 통한 ChatGPT의 BIM 분야 전문 지식 수준 평가

  • 최지원 (서울과학기술대학교 건설시스템공학과) ;
  • 구본상 (서울과학기술대학교 건설시스템공학과) ;
  • 유영수 (서울과학기술대학교 건설시스템공학과) ;
  • 정유정 (서울과학기술대학교 건설시스템공학과) ;
  • 함남혁 (한양사이버대학교 디지털건축도시공학과)
  • Received : 2023.06.16
  • Accepted : 2023.09.10
  • Published : 2023.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

ChatGPT, a chatbot based on GPT large language models, has gained immense popularity among the general public as well as domain professionals. To assess its proficiency in specialized fields, ChatGPT was tested on mainstream exams like the bar exam and medical licensing tests. This study evaluated ChatGPT's ability to answer questions related to Building Information Modeling (BIM) by testing it on Korea's BIM expertise exam, focusing primarily on multiple-choice problems. Both GPT-3.5 and GPT-4 were tested by prompting them to provide the correct answers to three years' worth of exams, totaling 150 questions. The results showed that both versions passed the test with average scores of 68 and 85, respectively. GPT-4 performed particularly well in categories related to 'BIM software' and 'Smart Construction technology'. However, it did not fare well in 'BIM applications'. Both versions were more proficient with short-answer choices than with sentence-length answers. Additionally, GPT-4 struggled with questions related to BIM policies and regulations specific to the Korean industry. Such limitations might be addressed by using tools like LangChain, which allow for feeding domain-specific documents to customize ChatGPT's responses. These advancements are anticipated to enhance ChatGPT's utility as a virtual assistant for BIM education and modeling automation.

Keywords

Acknowledgement

본 연구는 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(No. 2020R1A2C1100741).

References

  1. Beetz, J., Van Leeuwen, J., De Vries, B. (2009). IfcOWL : A Case of Transforming EXPRESS Schemas into Ontologies. Ai Edam, 23(1), pp. 89-101.  https://doi.org/10.1017/S0890060409000122
  2. Bommarito II, M., Katz, D. M. (2022). GPT Takes the Bar Exam. arXivpreprint arXiv:2212.14402. 
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G. Askell, A., Agarwal, S., Herbert-Voss, A. Krueger, G, Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. (2020).Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, pp. 1877-1901. 
  4. Choi, J. H., Hickman, K. E., Monahan, A., Schwarcz, D. (2023). ChatGPT Goes to Law School. Available at SSRN. 
  5. Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y., Radev, D. (2023). Evaluating GPT-4 and ChatGPT on Japanese-Medical Licensing Examinations. JMIR Medical Education, 9. 
  6. KIBIM (2023). A study on countermeasures for construction engineering in accordance with the digitalization of the construction industry, Final Report. 
  7. Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M., Aggabao, R., DiazCandido, G., Maningo, J., Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted Medical Education Using Large Language Models. PLOS Digital Health, 2(2). 
  8. Mazairac, W., Beetz, J. (2013). BIMQL-An Open Query language for Building Information Models. Advanced Engineering Informatics, 27(4), pp. 444-456.  https://doi.org/10.1016/j.aei.2013.06.001
  9. OpenAI (2023). GPT-4 Technical Report. ArXiv, abs/2303.08774. 
  10. Seo, M. B., Ju, K. B. (2012). Strategies to Revitalize BIM (Building Information Modeling) by the Survey Question-Naires from Design Experts in Field of Civil Engineering. Journal of the Korea Contents Association, 12(11), pp. 446-457.  https://doi.org/10.5392/JKCA.2012.12.11.446
  11. Singh, A. K., Pal, A., Kumar, P., Lin, J. J., Hsieh, S. H. (2023). Prospects of Integrating BIM and NLP for Automatic Construction Schedule Management. ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction. IAARC Publications, 40, pp. 238-245. 
  12. Zheng, J., Fischer, M. (2023). BIM-GPT: A Prompt-based Virtual Assistant Framework for BIM Information Retrieval. arXiv preprint, DOI: 10.48550/arXiv.2304.09333