Educational Technology International

The Korean Society for Education Technology (한국교육공학회)
권호 표지
DOI QR코드

DOI QR Code

Learner Perception of an Educational Recommender System based on Relative Importance of Learner Variables

  • Woorin HWANG (University of Florida) ;
  • Hyo-Jeong SO (Ewha Womans University)
  • Received : 2024.09.01
  • Accepted : 2024.10.21
  • Published : 2024.10.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study suggests that educational recommender systems should be explainable and extend beyond the commercially driven algorithms that primarily rely on user preferences and purchase behaviors. Instead, we propose a recommendation method that considers how and why people learn by employing the relative importance of various learner variables. To develop a recommendation algorithm, 100 adult participants used 4 to 6 foreign language learning mobile applications(apps), generating a dataset of 557 user perception reports. Using this data, we designed and developed a recommender system based on the importance weights of 14 learner variables, categorized into four groups: (a) demographic information, (b) motivational orientation for language learning (instrumental vs. integrative), (c) learning styles, and (d) learning experience. The results based on RandomForestRegressor model revealed that language learning motivation, learning styles (specifically information processing), and usage frequency were significantly more influential than general demographic factors in predicting learners' evaluation of the apps. Furthermore, learners' perception of the recommender system revealed that the recommender system was relevant and engaging, effectively meeting their needs and assisting them in selecting appropriate language learning apps. Overall, this study demonstrates the potential of educational recommender systems that consider learners' motivation, experience, and learning styles.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(NRF-2020R1F1A1073469). This article is based on research conducted as part of the first author's master's thesis at Ewha Womans University, with portions previously presented at the 2022 ICCE conference.

References

  1. Al-Taani, M. H. (2018). Integrative and instrumental motivations for learning English as a university requirement among undergraduate students at Al-Jazeera University. International Journal of Learning and Development, 8(4), 89-105.
  2. Bobadilla, J., Hernando, A., Ortega, F., & Gutierrez, A. (2012). Collaborative filtering based on significances. Information Sciences, 185(1), 1-17.
  3. Cheng, H. T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., ... & Shah, H. (2016, September). Wide & deep learning for recommender systems [Paper presentation]. The 1st workshop on deep learning for recommender systems (pp. 7-10). Boston, United States. https://doi.org/10.1145/2988450.2988454
  4. Davidsson, C., & Moritz, S. (2011, February). Utilizing implicit feedback and context to recommend mobile applications from first use [Paper presentation]. 2011 Workshop on Context-awareness in Retrieval and Recommendation (pp. 19-22). Palo Alto, CA, United States. https://doi.org/10.1145/1961634.1961639
  5. da Silva, F. L., Slodkowski, B. K., da Silva, K. K. A., & Cazella, S. C. (2023). A systematic literature review on educational recommender systems for teaching and learning: Research trends, limitations and opportunities. Education and Information Technologies, 28(3), 3289-3328. https://doi.org/10.1007/s10639-022-11341-9
  6. Dornyei, Z., & Kormos, J. (2000). The role of individual and social variables in oral task performance. Language Teaching Research, 4(3), 275-300.
  7. Essalmi, F., Ayed, L. J. B., Jemni, M., & Graf, S. (2015). Generalized metrics for the analysis of E-learning personalization strategies. Computers in Human Behavior, 48, 310-322.
  8. Felder, R. M., & Silverman, L. K. (1988). Learning and teaching styles in engineering education. Engineering Education, 78(7), 674-681.
  9. Felder, R. M., & Solomon, B. A. (1996)., Index of Learning Styles Questionnaire. https://www.webtools.ncsu.edu/learningstyles/
  10. Fernandez-Garcia, A. J., Rodriguez-Echeverria, R., Preciado, J. C., Manzano, J. M. C., & Sanchez-Figueroa, F. (2020). Creating a recommender system to support higher education students in the subject enrollment decision. IEEE Access, 8, 189069-189088. https://doi.org/10.1109/ACCESS.2020.3031572
  11. Garcia-Martinez, S., & Hamou-Lhadj, A. (2013). Educational recommender systems: A pedagogical-focused perspective. In G. A. Tsihrintzis, M. Virvou, & L. C. Jain (Eds.), Multimedia services in intelligent environments: Recommendation services (Vol. 25, pp. 113-124). Springer, Heidelberg. https://doi.org/10.1007/978-3-319-00375-7_8
  12. Genuer, R., Poggi, J. M., & Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225-2236.
  13. Geron, A. (2019). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O'Reilly Media, Inc.
  14. Gromping, U. (2009). Variable importance assessment in regression: linear regression versus random forest. The American Statistician, 63(4), 308-319.
  15. Han, J. & Lim, H. S. (2017, August 10-11) Analysis of the effectiveness of the recommendation model for the customized learning course [Conference presentation]. The Korean Association of Computer Education 2017 Annual Conference (pp. 221-224), Daegu, Republic of Korea
  16. Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education promises and implications for teaching and learning. Center for Curriculum Redesign.
  17. Huang, L., Wang, C.-D., Chao, H.-Y., Lai, J.-H., & Yu, P. S. (2019). A score prediction approach for optional course recommendation via cross-userdomain collaborative filtering. IEEE Access, 7, 19550-19563. https:// doi. org/ 10. 1109/ ACCESS. 2019. 28979 79 https://doi.org/10.1109/ACCESS.2019.2897979
  18. Ismail, H. M., Belkhouche, B., & Harous, S. (2019). Framework for personalized content recommendations to support informal learning in massively diverse information Wikis. IEEE Access, 7, 172752-172773. https://doi.org/10.1109/ACCESS.2019.29562 84
  19. Jung, S., (2019) A case study of college students' use of mobile applications and their perception of AI-based Apps. The Journal of Mirae English Language and Literature, 24(3), 303-330.
  20. Klasnja-Milicevic, A., Vesin, B., & Ivanovic, M. (2018). Social tagging strategy for enhancing e-learning experience. Computers & Education, 118, 166-181. https:// doi. org/ 10. 1016/j. compe du. 2017. 12. 002 https://doi.org/10.1016/j.compedu.2017.12.002
  21. Lee, H., So, H., & Jin, L. (2020). Analysis of the characteristics and design level of educational artificial intelligence applications. Journal of Korean Association for Educational Information and Media, 26(3), 647-670.
  22. Lightbown, P. M., & Spada, N. (2021). How languages are learned 5th Edition. Oxford University Press.
  23. Nabizadeh, A. H., Goncalves, D., Gama, S., Jorge, J., & Rafsanjani, H. N. (2020). Adaptive learning path recommender approach using auxiliary learning objects. Computers & Education, 147, 103777-103793. https:// doi. org/ 10. 1016/j. compe du. 2019. 103777 https://doi.org/10.1016/j.compedu.2019.103777
  24. Nafea, S. M., Siewe, F., & He, Y. (2019). On Recommendation of learning objects using Felder-Silverman learning style model. IEEE Access, 7, 163034-163048. https:// doi. org/ 10. 1109/ ACCESS. 2019. 2935417 https://doi.org/10.1109/ACCESS.2019.2935417
  25. Pesovski, I., Bogdanova, A. M., & Trajkovik, V. (2022, November). Systematic review of the published explainable educational recommendation systems. 2022 20th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1-8). Antalya, Turkey, https://doi.org/10.1109/ITHET56107.2022.10032029
  26. Pimenidis, E., Polatidis, N., & Mouratidis, H. (2018). Mobile recommender systems: Identifying the major concepts. Journal of Information Science, 45, 387-397. https://doi.org/10.1177/0165551518792213
  27. Premlatha, K. R., Dharani, B., & Geetha, T. V. (2016). Dynamic learner profiling and automatic learner classification for adaptive e-learning environment. Interactive Learning Environments, 24(6), 1054-1075.
  28. Qiao, C. (2024). Factors influencing second language learning based on the research of Lightbown and Spada. Frontiers in Psychology, 15, https://doi.org/10.3389/fpsyg.2024.1347691
  29. Rahman, M. M., & Abdullah, N. A. (2018). A personalized group-based recommendation approach for web search in E-Learning. IEEE Access, 6, 34166-34178. https:// doi. org/ 10. 1109/ ACCESS. 2018.28503 76 https://doi.org/10.1109/ACCESS.2018.2850376
  30. Ricci, F., Rokach, L., & Shapira, B. (2015). Recommender systems: introduction and challenges. In F. Ricci, L. Rokach, B. Shapira, & P. B. Kantor (Eds.), Recommender systems handbook (pp. 1-34). Springer. https://doi.org/10.1007/978-1-4899-7637-6_1.
  31. Santos, O. C., & Boticario, J. G. (2012). Educational recommender systems and technologies: Practices and challenges. IGI Global.
  32. Scholz, M., Dorner, V., Schryen, G., & Benlian, A. (2017). A configuration-based recommender system for supporting e-commerce decisions. European Journal of Operational Research, 259(1), 205-215.
  33. Sergis, S., & Sampson, D. G. (2016). Learning object recommendations for teachers based on elicited ICT competence profiles. IEEE Transactions on Learning Technologies, 9(1), 67-80. https:// doi. org/ 10.1109/ TLT. 2015. 24348 24
  34. Tarus, J. K., Niu, Z., & Yousif, A. (2017). A hybrid knowledge-based recommender system for e-learning based on ontology and sequential pattern mining. Future Generation Computer Systems, 72, 37-48. https://doi.org/10.1016/j.future.2017.02.049
  35. Vultureanu-Albisi, A., & Badica, C. (2021). Recommender systems: an explainable AI perspective. 2021 International Conference on Innovations in Intelligent Systems and Applications (INISTA) ( pp. 1-6). https://doi.org/10.1109/INISTA52262.2021.9548125
  36. Wan, S., & Niu, Z. (2016). A learner oriented learning recommendation approach based on mixed concept mapping and immune algorithm. Knowledge-Based Systems, 103, 28-40. https:// doi. org/ 10. 1016/j. knosys.2016. 03. 022 https://doi.org/10.1016/j.knosys.2016.03.022
  37. Wan, S., & Niu, Z. (2018). An e-learning recommendation approach based on the self-organization of learning resource. Knowledge-Based Systems, 160, 71-87. https:// doi. org/ 10. 1016/j. knosys. 2018. 06.014 https://doi.org/10.1016/j.knosys.2018.06.014
  38. Wan, S., & Niu, Z. (2020). A hybrid E-Learning recommendation approach based on learners' influence propagation. IEEE Transactions on Knowledge and Data Engineering, 32(5), 827-840. https://doi.org/10.1109/TKDE.2019.2895033
  39. Williamson, B. D., Gilbert, P. B., Simon, N. R., & Carone, M. (2023). A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association, 118(543), 1645-1658.
  40. Wu, D., Lu, J., & Zhang, G. (2015). A Fuzzy Tree Matching-based personalized ELearning recommender system. IEEE Transactions on Fuzzy Systems, 23(6), 2412-2426. https:// doi. org/ 10. 1109/ TFUZZ. 2015. 2426201 https://doi.org/10.1109/TFUZZ.2015.2426201
  41. Wu, Z., Li, M., Tang, Y., & Liang, Q. (2020). Exercise recommendation based on knowledge concept prediction. Knowledge-Based Systems, 210, 106481-106492. https:// doi. org/ 10. 1016/j. knosys. 2020.106481 https://doi.org/10.1016/j.knosys.2020.106481
  42. Yanes, N., Mostafa, A. M., Ezz, M., & Almuayqil, S. N. (2020). A machine learningbased recommender system for improving students learning experiences. IEEE Access, 8, 201218-201235. https://doi.org/10.1109/ACCESS.2020.3036336
  43. Yuen, C. L., & Schlote, N. (2024). Learner experiences of mobile apps and artificial intelligence to support additional language learning in education. Journal of Educational Technology Systems, 52 (4) https://doi.org/10.1177/00472395241238693
  44. Zapata, A., Menendez, V. H., Prieto, M. E., & Romero, C. (2015). Evaluation and selection of group recommendation strategies for collaborative searching of learning objects. International Journal of Human-Computer Studies, 76, 22-39. https:// doi. org/ 10. 1016/j. ijhcs. 2014. 12. 002 https://doi.org/10.1016/j.ijhcs.2014.12.002
  45. Zhang, Y., & Chen, X. (2020). Explainable recommendation: A survey and new perspectives. Foundations and Trends® in Information Retrieval, 14(1), 1-101