Understanding Topical Relevance of Multimedia based on EEG Techniques

뇌파측정기술(EEG)에 기초한 멀티미디어 자료의 주제 적합성에 관한 연구

  • 김현희 (명지대학교 문헌정보학과) ;
  • 김용호 (부경대학교 신문방송학과)
  • Received : 2016.07.25
  • Accepted : 2016.08.18
  • Published : 2016.08.30


This study proposed two topical relevance models, simple and complex models, using EEG/ERP techniques. In the simple model regarding simple search tasks, N300 and P3b components are used. The N300 is specific to the semantic processing of pictures and the P3b reflects mechanisms involved in the decision about whether an external stimulus matches or does not match an internal representation of a specific category. In the complex model regarding complex search tasks, on the other hand, N400 and P600 components are used. The N400 reflects activation of an amodel system that integrates both image-based and conceptual representations into a context, whereas the P600 is related to complex cognitive processes. Our research results can be used as a source to design an EEG-based interactive multimedia system.


EEG;ERP;Simple Topical-relevance Model;Complex Topical-relevance Model;Video;Relevance


  1. 김현희. 2015. 뇌파측정기술(EEG)과 판별분석을 이용한 영상물의 키프레임 자동 분류 방안 연구. 정보관리학회지, 32(3): 377-396. Kim, Hyun-Hee. 2015. "Toward a Key-Frame Automatic Extraction Method for Video StoryboardSurrogates based on Users' EEG Signals and Discriminant Analysis." Journal of the KoreanSociety for Information Management, 32(3): 377-396.
  2. 박명관, 나윤주. 2012. 통사-의미 역학의 신경언어학적 연구. 생성문법연구, 22(1): 157-183. Park, Myuung-Kwan and Na, Yoonjoo. 2012. "A Neurolinguistic Study of Syntax-SemanticsDynamics." Studies in Generative Grammar, 22(1): 157-183.
  3. Allegretti, M. et al. 2015. "When Relevance Judgement Is Happening?: An EEG-based Study." In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2015), August 9-13th, 2015, Santiago, Chile:PUC Extension Center: 719-722.
  4. Baddeley, A. 2007. Working Memory, Thought and Action. Oxford: Oxford University Press.
  5. Barrett, S. and Rugg, M. 1990. "Event-Related Potentials and the Semantic Matching of Pictures." Brain and Cognition, 14(2): 201-212.
  6. Graben, P. B., Gerth, S. and Vasishth, S. 2008. "Towards Dynamical System Models of Language-Related Brain Potentials." Cognitive Neurodynamics, 2(3): 229-55.
  7. Chen, Z. and Xu, Y. 2005. "User-Oriented Relevance Judgment: A Conceptual Model." In Proceedings of the 38th Hawaii International Conference on System Sciences, January 6th, 2005, Big Island, HI: Hilton Waikoloa Village Resort: 1-11.
  8. Choi, Y. and Rasmussen, E. 2002. Users' Relevance Criteria in Image Retrieval in American History. Information Processing & Management, 38(5): 695-726.
  9. Cosijna, E. and Ingwersena, P. 2000. "Dimensions of Relevance." Information Processing & Management, 36(4): 533-550.
  10. Corbetta, M. and Shulman, G. 2002. "Control of Goal-Directed and Stimulus-Driven Attention in the Brain." Nature Reviews Neuroscience, 3(3): 201-215.
  11. Donchin, E. 1981. "Surprise!...Surprise?." Psychophysiology, 18(5): 493-513.
  12. Donchin, E. and Coles, M. G. 1988. "Is the P300 Component a Manifestation of Context Updating." Behavioral and Brain Sciences, 11(3): 357-427.
  13. Eugster, M. et al. 2014. "Predicting Term-Relevance from Brain Signals." In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR 2014). July 6-11th, 2014, Broadbeach, QLD: Gold Coast Convention and Exhibition Centre: 425-434.
  14. Federmeier, K. D. and Laszlo, S. 2009. "Time for Meaning: Electrophysiology Provides Insights into the Dynamics of Representation and Processing in Semantic Memory." In B. H. Ross. Psychology of Learning and Motivation, 51. Burlington: Academic Press, 1-44.
  15. Friederici, A. D. 1995. "The Time Course of Syntactic Activation during Language Processing:A Model based on Neuropsychological and Neurophysiological Data." Brain and Language, 50: 259-281.
  16. Harter, S. P. 1992. "Psychological Relevance and Information Science." Journal of the American Society for Information Science and Technology, 43(9): 602-615.<602::AID-ASI3>3.0.CO;2-Q
  17. Huang, X. and Soergel, D. 2006. "An Evidence Perspective on Topical Relevance Types and Its Implications for Exploratory and Task-based Retrieval." Information Research, 12(1):281.
  18. Kaan, E. and Swaab, T. 2003. "Repair, Revision, and Complexity in Syntactic Analysis:An Electrophysiological Differentiation." Journal of Cognitive Neuroscience, 15(1): 98-110.
  19. Kim, A. and Sikos, L. 2011. "Conflict and Surrender during Sentence Processing: An ERP Study of Syntax-Semantics Interaction." Brain and Language, 118(1-2): 15-22.
  20. Koelstra, S., Muehl, C. and Patras, I. 2009. "EEG Analysis for Implicit Tagging of Video Data." In Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII. September 10-12th, 2009, Amsterdam: 1-14.
  21. Kok, A. 1997. "Event-Related-Potential (ERP) Reflections of Mental Resources: A Review and Synthesis." Biological Psychology, 45: 19-56.
  22. Kok, A. 2001. "On the Utility of P3 Amplitude as a Measure of Processing Capacity." Psychophysiology, 38: 557-577.
  23. Komaki, D., Hara, T. and Nishio, S. 2012. "How Does Mobile Context Affect People's Web Search Behavior?: A Diary Study of Mobile Information Needs and Search Behaviors." In Proceedings of 2012 IEEE 26th International Conference on Advanced Information Networking and Applications, March 26-29th, 2012, Fukuoka: Fukuoka Institute of Technology: 245-252.
  24. Kuperberg, G. et al. 2007. "The Role of Animacy and Thematic Relationships in Processing Active English Sentences: Evidence from Event-Related Potentials." Brain and Language, 100(3): 223-237.
  25. Kutas, M. and Federmeier, K. D. 2011. "Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP)." Annual Review of Psychology, 62: 621-647.
  26. Kutas, M. and Hillyard, S. A. 1980. "Reading Senseless Sentences: Brain Potentials Reflect Semantic Incongruity." Science, 207: 203-208.
  27. Maillard, L. et al. 2011. "From Perception to Recognition Memory: Time Course and Lateralization of Neural Substrates of Word and Abstract Picture Processing." Journal of Cognitive Neuroscience, 23(4): 782-800.
  28. McPherson, W. and Holcomb, P. 1999. "An Electrophysiological Investigation of Semantic Priming with Pictures of Real Objects." Psychophysiology, 36: 53-65.
  29. Mehravari, A. S. et al. 2015. "Effects of Grammaticality and Morphological Complexity on the P600 Event-Related Potential Component." PLoS ONE, 10(10): 1-16.
  30. Mizzaro, S. 1997. "Relevance: The Whole History." Journal of the American Society for Information Science and Technology, 48(9): 810-832.<810::AID-ASI6>3.0.CO;2-U
  31. Moshfeghi, Y. et al. 2013. "Understanding Relevance: An fMRI Study." In Serdyukov, P. et al. Advances in Information Retrieval. Heidelberg: Springer.
  32. Mostafa, J. and Gwizdka, J. 2016. "Deepening the Role of the User: Neuro-Physiological Evidence as a Basis for Studying and Improving Search." In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. March 13-17th, 2016, North Carolina: Hampton Inn and Suites Chapel Hill-Carrboro: 63-70.
  33. Neville, H. et al. 1986. "Event-Related Brain Potentials during Initial Encoding and Recognition Memory of Congruous and Incongruous Words." Journal of Memory and Language, 25:75-92.
  34. Osterhout, L. and Holcomb, P. 1992. "Event-Related Brain Potentials Elicited by Syntactic Anomaly." Journal of Memory and Language, 31(6): 785-806.
  35. Polich, J. 2007. "Updating P300: An Integrative Theory of P3a and P3b." Clinical Neurophysiology, 118(10): 2128-2148.
  36. Saracevic, T. 1996. Relevance Reconsidered. In Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS2). October 14-17rh, 1996, Copenhagen:Royal School of Librarianship: 201-218.
  37. Saracevic, T. 2007. "Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science: Part II: Nature and Manifestations of Relevance." Journal of the American Society for Information Science and Technology, 58(13): 1915-1933.
  38. Singer, G., Danilov, D. and Norbisrath, U. 2012. "Complex Search: Aggregation, Discovery, and Synthesis." Proceedings of the Estonian Academy of Sciences, 61(2): 89-106.
  39. Singer G., Norbisrath, U. and Lewandowski, D. 2013. "Ordinary Search Engine Users Carrying Out Complex Search Tasks." Journal of Information Science, 39(3): 346-358.
  40. Tang, R. and Solomon, P. 1998. "Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior." Information Processing and Management, 34(2/3): 237-256.
  41. van de Meerendonk, N. et al. 2010. "Monitoring in Language Perception: Mild and Strong Conflicts Elicit Different ERP Patterns." Journal of Cognitive Neuroscience, 22(1): 67-82.
  42. Verleger, R. 1988. "Event-Related Potentials and Cognition: A Critique of the Context Updating Hypothesis and an Alternative Interpretation of P3." Behavioral and Brain Sciences, 11: 343-356.
  43. Verleger, R., Jaskowski, P. and Wascher, E. 2005. "Evidence for an Integrative Role of P3b in Linking Reaction to Perception." Journal of Psychophysiology, 19: 165-181.
  44. Wagner, A. et al. 1998. "Experiences as Predicted by Brain Activity Building." Science, 281(5380): 1188-1191.
  45. West, W. and Holcomb, P. 2002. "Event-Related Potentials during Discourse-Level Semantic Integration of Complex Pictures." Cognitive Brain Research, 13: 363-375.