Journal of the Korean Society for information Management (정보관리학회지)

Korean Society for Information Management (한국정보관리학회)
권호 표지
DOI QR코드

DOI QR Code

Digital Library Interface Research Based on EEG, Eye-Tracking, and Artificial Intelligence Technologies: Focusing on the Utilization of Implicit Relevance Feedback

뇌파, 시선추적 및 인공지능 기술에 기반한 디지털 도서관 인터페이스 연구: 암묵적 적합성 피드백 활용을 중심으로

  • 김현희 (명지대학교 문헌정보학과) ;
  • 김용호 (부경대학교 미디어커뮤니케이션학부)
  • Received : 2024.02.15
  • Accepted : 2024.03.04
  • Published : 2024.03.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study proposed and evaluated electroencephalography (EEG)-based and eye-tracking-based methods to determine relevance by utilizing users' implicit relevance feedback while navigating content in a digital library. For this, EEG/eye-tracking experiments were conducted on 32 participants using video, image, and text data. To assess the usefulness of the proposed methods, deep learning-based artificial intelligence (AI) techniques were used as a competitive benchmark. The evaluation results showed that EEG component-based methods (av_P600 and f_P3b components) demonstrated high classification accuracy in selecting relevant videos and images (faces/emotions). In contrast, AI-based methods, specifically object recognition and natural language processing, showed high classification accuracy for selecting images (objects) and texts (newspaper articles). Finally, guidelines for implementing a digital library interface based on EEG, eye-tracking, and artificial intelligence technologies have been proposed. Specifically, a system model based on implicit relevance feedback has been presented. Moreover, to enhance classification accuracy, methods suitable for each media type have been suggested, including EEG-based, eye-tracking-based, and AI-based approaches.

본 연구는 디지털 도서관의 콘텐츠를 탐색하는 동안 이용자의 암묵적 적합성 피드백을 활용하여 적합성을 판단하기 위해 뇌파 기반 및 시선추적 기반 방법들을 제안하고 평가해 보았다. 이를 위해서 32명을 대상으로 하여 동영상, 이미지, 텍스트 데이터를 활용하여 뇌파/시선추적 실험들을 수행하였다. 제안된 방법들의 유용성을 평가하기 위해서, 딥러닝 기반의 인공지능 방법들을 경쟁 기준으로 사용하였다. 평가 결과, 주제에 적합한 동영상과 이미지(얼굴/감정)를 선택하는 데에는 뇌파 컴포넌트 기반 방법들(av_P600, f_P3b)이 높은 분류 정확도를 나타냈고, 이미지(객체)와 텍스트(신문 기사)를 선택하는 데에는 인공지능 기반 방법 즉, 객체 인식 기반 방법과 자연언어 처리 방법이 각각 높은 분류 정확도를 나타냈다. 끝으로, 뇌파, 시선추적 및 인공지능 기술에 기반한 디지털 도서관 인터페이스를 구현하기 위한 지침 즉, 암묵적 적합성 피드백에 기반한 시스템 모형을 제안하고, 분류 정확도를 향상시키기 위해서 미디어별로 적합한 뇌파 기반, 시선추적 기반 및 인공지능 기반 방법들을 제시하였다.

Keywords

Acknowledgement

이 논문은 2020년 대한민국 교육부와 한국연구재단의 인문사회분야 중견연구자지원사업의 지원을 받아 수행된 연구임(NRF-2020S1A5A2A01040945).

References

  1. Yoon, Jung Won & Syn, Sue Yeon (2021). How do formats of health related Facebook posts effect on eye movements and cognitive outcomes?. Journal of the Korean Society for Library and Information Science, 55(3), 219-237. http://doi.org/10.4275/KSLIS.2021.55.3.219
  2. Allegretti, M., Moshfeghi, Y., Hadjigeorgieva, M., Pollick, F. E., Jose, J. M., & Pasi, G. (2015). When relevance judgement is happening? An EEG-based study. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 719-722. https://doi.org/10.1145/2766462.2767811
  3. Bhattacharya, N., Rakshit, S., Gwizdka, J., & Kogut, P. (2020). Relevance prediction from eye-movements using semi-interpretable convolutional neural networks. Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, 223-233. https://doi.org/10.1145/3343413.3377960
  4. Borgalli, M. R. A. & Surve, S. (2022). Deep learning for facial emotion recognition using custom CNN architecture. Journal of Physics: Conference Series, 2236(1), 012004. https://doi.org/10.1088/1742-6596/2236/1/012004
  5. Davis, K. M., Spape, M., & Ruotsalo, T. (2022). Contradicted by the brain: Predicting individual and group preferences via brain-computer interfacing. IEEE Transactions on Affective Computing, 14(4), 3094-3105. https://doi.org/10.1109/TAFFC.2022.3225885
  6. DeLong, K. A., Quante, L., & Kutas, M. (2014). Predictability, plausibility, and two late ERP positivities during written sentence comprehension. Neuropsychologia, 61, 150-162. https://doi.org/10.1016/j.neuropsychologia.2014.06.016
  7. Eugster, M. J., Ruotsalo, T., Spape, M. M., Barral, O., Ravaja, N., Jacucci, G., & Kaski, S. (2016). Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals. Scientific Reports, 6(1), 38580. https://doi.org/10.1038/srep38580
  8. Eugster, M. J., Ruotsalo, T., Spape, M. M., Kosunen, I., Barral, O., Ravaja, N., Jacucci, G., & Kaski, S. (2014). Predicting term-relevance from brain signals. Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, 425-434. https://doi.org/10.1145/2600428.2609594
  9. Evans, W. J., Cui, L., & Starr, A. (1995). Olfactory event-related potentials in normal human subjects: Effects of age and gender. Electroencephalography and Clinical Neurophysiology, 95(4), 293-301. https://doi.org/10.1016/0013-4694(95)00055-4
  10. Foley, J. J. & Kwan, P. (2015). Feature extraction in content-based image retrieval. In Khosrow-Pour, D. B. A. ed. Encyclopedia of Information Science and Technology, Third Edition. Pennsylvania: IGI Global.
  11. Golenia, J. E., Wenzel, M. A., Bogojeski, M., & Blankertz, B. (2018). Implicit relevance feedback from electroencephalography and eye tracking in image search. Journal of Neural Engineering, 15(2), 026002. http://dx.doi.org/10.1088/1741-2552/aa9999
  12. Gwizdka, J. & Zhang, Y. (2015). Differences in eye-tracking measures between visits and revisits to relevant and irrelevant web pages. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 811-814. https://doi.org/10.1145/2766462.2767795
  13. Gwizdka, J., Hosseini, R., Cole, M., & Wang, S. (2017). Temporal dynamics of eye-tracking and EEG during reading and relevance decisions. Journal of the Association for Information Science and Technology, 68(10), 2299-2312. https://doi.org/10.1002/asi.23904
  14. Jacucci, G., Barral, O., Daee, P., Wenzel, M., Serim, B., Ruotsalo, T., Pluchino, P., Freeman J., Gamberini, L., Kaski, S., & Blankertz, B. (2019). Integrating neurophysiologic relevance feedback in intent modeling for information retrieval. Journal of the Association for Information Science and Technology, 70(9), 917-930. https://doi.org/10.1002/asi.24161
  15. Kim, H. H. & Kim, Y. H. (2019a). Video summarization using event-related potential responses to shot boundaries in real-time video watching. Journal of the Association for Information Science and Technology, 70(2), 164-175. https://doi.org/10.1002/asi.24103
  16. Kim, H. H. & Kim, Y. H. (2019b). ERP/MMR algorithm for classifying topic-relevant and topic-irrelevant visual shots of documentary videos. Journal of the Association for Information Science and Technology, 70(9), 931-941. https://doi.org/10.1002/asi.24179
  17. Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer ed. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press, 134-146.
  18. Pold, T., Bachmann, M., Paeske, L., Kalev, K., Lass, J., & Hinrikus, H. (2018). EEG spectral asymmetry is dependent on education level of men. World Congress on Medical Physics and Biomedical Engineering 2018, 2, 405-408. https://doi.org/10.1007/978-981-10-9038-7_76
  19. Schindler, S. & Bublatzky, F. (2020). Attention and emotion: An integrative review of emotional face processing as a function of attention. Cortex, 130, 362-382. https://doi.org/10.1016/j.cortex.2020.06.010
  20. Shi, Z. F., Zhou, C., Zheng, W. L., & Lu, B. L. (2017). Attention evaluation with eye tracking glasses for EEG-based emotion recognition. 8th International IEEE/EMBS Conference on Neural Engineering (NER), 86-89. https://doi.org/10.1109/NER.2017.8008298
  21. Udurume, M., Valverde, E. C., Caliwag, A., Kim, S., & Lim, W. (2023). Real-time multimodal emotion recognition based on multithreaded weighted average fusion. Journal of the Ergonomics Society of Korea, 42(5), 417-433. http://doi.org/ 10.5143/JESK.2023.42.5.417
  22. Ul Haq, H. B., Asif, M., Ahmad, M. B., Ashraf, R., & Mahmood, T. (2022). An effective video summarization framework based on the object of interest using deep learning. Mathematical Problems in Engineering, 2022, 7453744. https://doi.org/10.1155/2022/7453744
  23. Wang, W., Song, H., Zhao, S., Shen, J., Zhao, S., Hoi, S. C., & Ling, H. (2019). Learning unsupervised video object segmentation through visual attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3064-3074.
  24. Ye, Z., Xie, X., Liu, Y., Wang, Z., Chen, X., Zhang, M., & Ma, S. (2022). Towards a better understanding of human reading comprehension with brain signals. Proceedings of the ACM Web Conference, 380-391. https://doi.org/10.1145/3485447.3511966
  25. Zheng, W. L., Liu, W., Lu, Y., Lu, B. L., & Cichocki, A. (2018). Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, 49(3), 1110-1122. http://dx.doi.org/10.1109/TCYB.2018.2797176