Knowledge Management Research (지식경영연구)

The Knowledge Management Society of Korea (한국지식경영학회)
권호 표지
DOI QR코드

DOI QR Code

Development and Validation of a Korean Generative AI Literacy Scale

한국형 생성 인공지능 리터러시 척도 개발 및 타당화

  • Hwan-Ho Noh (Barun ICT Research Center, Yonsei University) ;
  • Hyeonjeong Kim (Barun ICT Research Center, Yonsei University) ;
  • Minjin Kim (Barun ICT Research Center, Yonsei University)
  • 노환호 (연세대학교 바른ICT연구소) ;
  • 김현정 (연세대학교 바른ICT연구소) ;
  • 김민진 (연세대학교 바른ICT연구소)
  • Received : 2024.07.15
  • Accepted : 2024.09.19
  • Published : 2024.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Literacy initially referred to the ability to read and understand written documents and processed information. With the advancement of digital technology, the scope of literacy expanded to include the access and use of digital information, evolving into the concept of digital literacy. The application and purpose of digital literacy vary across different fields, leading to the use of various terminologies. This study focuses on generative artificial intelligence (AI), which is gaining increasing importance in the AI era, to assess users' literacy levels. The research aimed to extend the concept of literacy proposed in previous studies and develop a tool suitable for Korean users. Through exploratory factor analysis, we identified that generative AI literacy consists of four factors: AI utilization ability, critical evaluation, ethical use, and creative application. Subsequently, confirmatory factor analysis validated the statistical appropriateness of the model structure composed of these four factors. Additionally, correlation analyses between the newly developed literacy tool and existing AI literacy scales and AI service evaluation tools revealed significant relationships, confirming the validity of the tool. Finally, the implications, limitations, and directions for future research are discussed.

리터러시는 작성된 문서나 가공된 정보를 사람들이 읽고 이해할 수 있는 능력으로 시작되었다. 이후 디지털 기술이 발전하면서 정보를 접하고 활용할 수 있는 분야가 넓어짐에 따라 디지털 리터러시로 확장되었다. 디지털 리터러시의 활용 분야와 목적에 따라 다양한 용어로 사용되고 있다. 본 연구에서는 인공지능 시대에 그 중요성이 점차 높아지고 있는 생성 인공지능을 대상으로 이용자의 리터러시 수준을 확인하고, 선행 연구에서 제안한 리터러시의 개념을 확장하며, 한국 이용자에게 적합한 도구를 개발하고자 연구를 수행했다. 먼저 탐색적 요인 분석 결과, 생성 인공지능 리터러시는 AI 활용 능력, 비판적 평가, 윤리적 사용, 창의적 활용의 네 가지 요인으로 구성됨을 확인하였다. 다음으로 확인적 요인 분석을 통해 생성 인공지능 리터러시를 구성하는 네 가지 요인의 모형 구조가 통계적으로 적합하다는 것을 확인했다. 더불어 기존 인공지능 리터러시 척도와 인공지능 서비스 관련 평가 도구와의 상관 분석을 통해 새로 개발된 리터러시 도구가 실제 사람들의 태도와 유의한 관계가 있음을 확인하였고, 이를 통해 타당성이 확보되었음을 알 수 있었다. 마지막으로 본 연구의 의의와 한계점, 그리고 향후 연구 방향에 대해 논했다.

Keywords

References

  1. 김세나, 표두미, 이지선, 이준기, 신강현, 김경일 (2015). 한국판 조절초점 척도 타당화 연구. 한국심리학회지: 사회 및 성격, 29(3), 85-110. 
  2. 류미영, 한선관 (2023). 인공지능 리터러시 측정을 위한 검사 지표 개발. 한국컴퓨터정보학회논문지, 28(7), 39-46. 
  3. 이경주, 김은영 (2020). 플랫폼 서비스 혁신에 있어 인공지능(AI) 의 역할과 효과에 관한 연구: 카카오 그룹의 인공지능 활용 사례 연구. 지식경영연구, 21(1), 175-195. 
  4. 이병관, 윤태웅, 노환호, 임혜빈 (2019). 한국형 사회적 배제 경험 척도 개발 및 타당화 연구. 한국심리학회지: 소비자.광고, 20(1), 127-152. 
  5. 지유첸, 백지연, 조현정, 이지연 (2022). 사무직 직원의 디지털 리터러시와 무형식 학습의 관계에서 인센티브의 조절효과에 대한 연구: 중국의 사례를 중심으로. 지식경영연구, 23(3), 173-192. 
  6. 최숙영 (2022). AI 리터러시 프레임워크에 대한 연구. 컴퓨터교육학회 논문지, 25(5), 73-84. 
  7. 탁진국 (2007). 심리검사 -개발과 평가방법의 이해- (2판). 학지사. 
  8. 황용석, 이선민, 김여립, 황현정 (2022). 디지털 역량 척도 개발 연구. 언론정보연구, 59(2), 5-48.
  9. Acharya, N., Sassenberg, A. M., & Soar, J. (2023). Effects of cognitive absorption on continuous use intention of AI-driven recommender systems in e-commerce. Foresight, 25(2), 194-208. 
  10. Allcott, H., & Gentzkow, M. (2017). Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2), 211-36. 
  11. Ashok, M., Madan, R., Joha, A., & Sivarajah, U. (2022). Ethical framework for Artificial Intelligence and Digital technologies. International Journal of Information Management, 62, 102433. 
  12. Aydin, M. (2021). Does the digital divide matter? Factors and conditions that promote ICT literacy. Telematics and Informatics, 58, 101536. 
  13. Bago, B., Rand, D. G., & Pennycook, G. (2020). Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines. Journal of Experimental Psychology: General, 149(8), 1608-1613. 
  14. Bagozzi, R. P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74-94. 
  15. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238-246. 
  16. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588-606. 
  17. Brashier, N. M., & Marsh, E. J. (2020). Judging truth. Annual Review of Psychology, 71, 499-515. 
  18. Bronstein, M. V., Pennycook, G., Bear, A., Rand, D. G., & Cannon, T. D. (2019). Belief in fake news is associated with delusionality, dogmatism, religious fundamentalism, and reduced analytic thinking. Journal of Applied Research in Memory and Cognition, 8(1), 108-117. 
  19. Brown, J. A. (1998). Media literacy perspectives. Journal of Communication, 48(1), 44-57. 
  20. Buckingham, D., & Burn, A. (2007). Game literacy in theory and practice. Journal of Educational Multimedia and Hypermedia, 16(3), 323-349. 
  21. Butler, J. (2012). Grappling with change: Web 2.0 and teacher education. In D. Polly, C. Mims, & K. A. Persichitte (Eds.), Developing technology-rich teacher education programs: Key issues (pp. 135-150). Hershey, PA: IGI Global. 
  22. Cain, W. (2024). Prompting change: Exploring prompt engineering in large language model AI and its potential to transform education. TechTrends, 68(1), 47-57. 
  23. Carolus, A., Koch, M. J., Straka, S., Latoschik, M. E., & Wienrich, C. (2023). MAILS-Meta AI literacy scale: Development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change-and meta-competencies. Computers in Human Behavior: Artificial Humans, 1(2), 100014. 
  24. Celik, I. (2023). Exploring the determinants of artificial intelligence (Ai) literacy: Digital divide, computational thinking, cognitive absorption. Telematics and Informatics, 83, 102026. 
  25. Celik, I., Dindar, M., Muukkonen, H., & Jarvela, S. (2022). The promises and challenges of artificial intelligence for teachers: A systematic review of research. TechTrends, 66(4), 616-630. 
  26. Chai, C. S., Wang, X., & Xu, C. (2020). An extended theory of planned behavior for the modelling of Chinese secondary school students' intention to learn artificial intelligence. Mathematics, 8(11), 2089. 
  27. Chelli, M., Descamps, J., Lavoue, V., Trojani, C., Azar, M., Deckert, M., ... & Ruetsch-Chelli, C. (2024). Hallucination rates and reference accuracy of ChatGPT and bard for systematic reviews: Comparative analysis. Journal of Medical Internet Research, 26, e53164. 
  28. Chen, D., Wu, J., & Wang, Y. (2011). Unpacking new media literacy. Journal on Systemics, Cybernetics and Informatics, 9, 84-88. 
  29. Chi, O. H., Jia, S., Li, Y., & Gursoy, D. (2021). Developing a formative scale to measure consumers' trust toward interaction with artificially intelligent (AI) social robots in service delivery. Computers in Human Behavior, 118, 106700. 
  30. Davenport, T. H., & Ronanki, R. (2018). Artificial intelligence for the real world. Harvard Business Review, 96(1), 108-116. 
  31. DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Thousand Oaks, CA: Sage. 
  32. Duffy, B. (2018). The perils of perception. London, UK: Atlantic Books. 
  33. Dunn, A. G., Shih, I., Ayre, J., & Spallek, H. (2023). What generative AI means for trust in health communications. Journal of Communication in Healthcare, 16(4), 385-388. 
  34. Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research, 25(3), 277-304. 
  35. Ghallab, M. (2019). Responsible AI: Requirements and challenges. AI Perspectives, 1(1), 1-7. 
  36. Hagendorff, T. (2020). The ethics of AI ethics: An evaluation of guidelines. Minds and Machines, 30(1), 99-120. 
  37. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Englewood Cliffs: Prentice Hall. 
  38. Hetzel, R. D. (1996). A primer on factor analysis with comments of practices and reporting. In B. Thompson (Ed.), Advances in Social Science Methodology, Vol. 4. (pp.175-206). Greenwich, CT: JAL. 
  39. Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115-135. 
  40. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. 
  41. Jang, S. M., & Kim, J. K. (2018). Third person effects of fake news: Fake news regulation and media literacy interventions. Computers in Human Behavior, 80, 295-302. 
  42. Jiang, J., Karran, A. J., Coursaris, C. K., Leger, P. M., & Beringer, J. (2023). A situation awareness perspective on human-AI interaction: Tensions and opportunities. International Journal of Human-Computer Interaction, 39(9), 1789-1806. 
  43. Kang, E. Y. N., Chen, D. R., & Chen, Y. Y. (2023). Associations between literacy and attitudes toward artificial intelligence-assisted medical consultations: The mediating role of perceived distrust and efficiency of artificial intelligence. Computers in Human Behavior, 139, 107529. 
  44. Khasawneh, O. Y. (2018). Technophobia: Examining its hidden factors and defining it. Technology in Society, 54, 93-100. 
  45. Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). New York: Guilford Press. 
  46. Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence, 6, 100225. 
  47. Koc, M., & Barut, E. (2016). Development and validation of New Media Literacy Scale (NMLS) for university students. Computers in Human Behavior, 63, 834-843. 
  48. Kong, S. C., Cheung, W. M. Y., & Zhang, G. (2022). Evaluating artificial intelligence literacy courses for fostering conceptual learning, literacy and empowerment in university students: Refocusing to conceptual building. Computers in Human Behavior Reports, 7, 100223. 
  49. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121-1134. 
  50. Lin, P. Y., Chai, C. S., Jong, M. S. Y., Dai, Y., Guo, Y., & Qin, J. (2021). Modeling the structural relationship among primary students' motivation to learn artificial intelligence. Computers and Education: Artificial Intelligence, 2, 100006. 
  51. Lin, T. B., Li, J. Y., Deng, F., & Lee, L. (2013). Understanding new media literacy: An explorative theoretical framework. Journal of Educational Technology & Society, 16(4), 160-170. 
  52. List, A. (2019). Defining digital literacy development: An examination of pre-service teachers' beliefs. Computers & Education, 138, 146-158. 
  53. Livingstone, S., & Helsper, E. (2010). Balancing opportunities and risks in teenagers'use of the internet: The role of online skills and internet self-efficacy. New Media & Society, 12(2), 309-329. 
  54. Lo, L. S. (2023). The art and science of prompt engineering: A new literacy in the information age. Internet Reference Services Quarterly, 27(4), 203-210. 
  55. Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023). ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 74(5), 570-581. 
  56. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99. 
  57. Namkoong, J. E., & Henderson, M. D. (2016). Wanting a bird's eye to understand why: Motivated abstraction and causal uncertainty. Journal of Experimental Social Psychology, 64, 57-71. 
  58. Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, 100041. 
  59. Ng, D. T. K., Wu, W., Leung, J. K. L., Chiu, T. K. F., & Chu, S. K. W. (2024). Design and validation of the AI literacy questionnaire: The affective, behavioural, cognitive and ethical approach. British Journal of Educational Technology, 55(3), 1082-1104. 
  60. Nimon, K., Zigarmi, D., Houson, D., Witt, D., & Diehl, J. (2011). The work cognition inventory: Initial evidence of construct validity. Human Resource Development Quarterly, 22(1), 7-35. 
  61. Nunally, J. C. (1978). Psychometric Theory (2nd ed.). New York: McGraw-Hill. 
  62. Orru, G., Piarulli, A., Conversano, C., & Gemignani, A. (2023). Human-like problem-solving abilities in large language models using ChatGPT. Frontiers in Artificial Intelligence, 6, 1199350. 
  63. Pennycook, G., & Rand, D. G. (2019). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition, 188, 39-50. 
  64. Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7), 770-780. 
  65. Perry, K. H. (2012). What is literacy?--A critical overview of sociocultural perspectives. Journal of Language and Literacy Education, 8(1), 50-71. 
  66. Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125. 
  67. R Development Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. 
  68. Rajam, V., Reddy, A. B., & Banerjee, S. (2021). Explaining caste-based digital divide in India. Telematics and Informatics, 65, 101719. 
  69. Revelle, W. (2018). Psych: Procedures for psychological, psychometric, and personality research. R Package Version, 1(10). 
  70. Ritzer, G., & Jurgenson, N. (2010). Production, consumption, prosumption: The nature of capitalism in the age of the digital 'prosumer'. Journal of Consumer Culture, 10(1), 13-36. 
  71. Robinson, S. C. (2020). Trust, transparency, and openness: How inclusion of cultural values shapes Nordic national public policy strategies for artificial intelligence (AI). Technology in Society, 63, 101421. 
  72. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. 
  73. Saranto, K., & Hovenga, E. J. (2004). Information literacy-What it is about?: Literature review of the concept and the context. International Journal of Medical Informatics, 73(6), 503-513. 
  74. Simonsohn, U. (2009). Direct risk aversion: Evidence from risky prospects valued below their worst outcome. Psychological Science, 20(6), 686-692. 
  75. Southworth, J., Migliaccio, K., Glover, J., Reed, D., McCarty, C., Brendemuhl, J., & Thomas, A. (2023). Developing a model for AI Across the curriculum: Transforming the higher education landscape via innovation in AI literacy. Computers and Education: Artificial Intelligence, 4, 100127. 
  76. Staksrud, E., Olafsson, K., & Livingstone, S. (2013). Does the use of social networking sites increase children's risk of harm? Computers in Human Behavior, 29(1), 40-50. 
  77. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173-180. 
  78. Tan, A. S., Lee, C. J., & Chae, J. (2015). Exposure to health (mis) information: Lagged effects on young adults ㅍ health behaviors and potential pathways. Journal of Communication, 65(4), 674-698. 
  79. Tomarken, A. J., & Waller, N. G. (2005). Structural equation modeling: Strengths, limitations, and misconceptions. Annual Review of Clinical Psychology, 1, 31-65. 
  80. van der Zeeuw, A., Van Deursen, A. J., & Jansen, G. (2019). Inequalities in the social use of the Internet of things: A capital and skills perspective. New Media & Society, 21(6), 1344-1361. 
  81. van Laar, E., Van Deursen, A. J., Van Dijk, J. A., & De Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72, 577-588. 
  82. Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157-178. 
  83. Walter, Y. (2024). Embracing the future of Artificial Intelligence in the classroom: The relevance of AI literacy, prompt engineering, and critical thinking in modern education. International Journal of Educational Technology in Higher Education, 21(1), 15. 
  84. Wang, B., Rau, P. L. P., & Yuan, T. (2023). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324-1337. 
  85. Wang, C. H., & Wu, C. L. (2022). Bridging the digital divide: The smart TV as a platform for digital literacy among the elderly. Behaviour & Information Technology, 41(12), 2546-2559. 
  86. Wang, P. (2019). On defining artificial intelligence. Journal of Artificial General Intelligence, 10(2), 1-37. 
  87. Wang, W., & Siau, K. (2019). Artificial intelligence, machine learning, automation, robotics, future of work and future of humanity: A review and research agenda. Journal of Database Management (JDM), 30(1), 61-79. 
  88. Wang, Y. Y., & Wang, Y. S. (2022). Development and validation of an artificial intelligence anxiety scale: An initial application in predicting motivated learning behavior. Interactive Learning Environments, 30(4), 619-634. 
  89. White, A., Breazeale, M., & Collier, J. E. (2012). The effects of perceived fairness on customer responses to retailer SST push policies. Journal of Retailing, 88(2), 250-261.
  90. Zawacki-Richter, O., Marin, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education-where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1-27.
  91. 방송통신위원회 (2024). 2023년 지능정보사회 이용자 패널조사. https://www.kcc.go.kr/user.do;jsessionid=IAQBoWhhXJkTOg9Zt08sadcAu2FS8kBEgvfvp7Zc.servlet-aihgcldhome10?mode=view&page=A02060400&dc=K02060400&boardId=1030&cp=1&boardSeq=61938 
  92. OpenAI (2024). ChatGPT: An AI language model [Language model]. OpenAI, https://www.openai.com/research