Journal of the Korea Convergence Society (한국융합학회논문지)

Korea Convergence Society (한국융합학회)
권호 표지
DOI QR코드

DOI QR Code

Monitoring Mood Trends of Twitter Users using Multi-modal Analysis method of Texts and Images

텍스트 및 영상의 멀티모달분석을 이용한 트위터 사용자의 감성 흐름 모니터링 기술

  • Kim, Eun Yi (Visual Information Lab., Department of Software, Konkuk University) ;
  • Ko, Eunjeong (Visual Information Lab., Department of Software, Konkuk University)
  • 김은이 (건국대학교 소프트웨어학과 시각정보처리연구실) ;
  • 고은정 (건국대학교 소프트웨어학과 시각정보처리연구실)
  • Received : 2017.11.13
  • Accepted : 2018.01.20
  • Published : 2018.01.28
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a novel method for monitoring mood trend of Twitter users by analyzing their daily tweets for a long period. Then, to more accurately understand their tweets, we analyze all types of content in tweets, i.e., texts and emoticons, and images, thus develop a multimodal sentiment analysis method. In the proposed method, two single-modal analyses first are performed to extract the users' moods hidden in texts and images: a lexicon-based and learning-based text classifier and a learning-based image classifier. Thereafter, the extracted moods from the respective analyses are combined into a tweet mood and aggregated a daily mood. As a result, the proposed method generates a user daily mood flow graph, which allows us for monitoring the mood trend of users more intuitively. For evaluation, we perform two sets of experiment. First, we collect the data sets of 40,447 data. We evaluate our method via comparing the state-of-the-art techniques. In our experiments, we demonstrate that the proposed multimodal analysis method outperforms other baselines and our own methods using text-based tweets or images only. Furthermore, to evaluate the potential of the proposed method in monitoring users' mood trend, we tested the proposed method with 40 depressive users and 40 normal users. It proves that the proposed method can be effectively used in finding depressed users.

본 논문은 개인 사용자의 트윗을 분석하여 사용자의 감정 흐름을 모니터링할 수 있는 새로운 방법을 제안한다. 본 논문에서는 사용자의 감성 흐름을 정확하게 예측하기 위해서 기존의 텍스트 위주의 시스템과 달리 본 연구에서는 사용자가 쓴 텍스트와 영상 등으로부터 감성을 인식하는 멀티 모달 분석 기법이 개발된다. 제안된 방법에서는 먼저 어휘분석 및 문맥을 이용한 텍스트분석기와 학습기반의 영상감성인식기를 이용하여 텍스트 및 영상 트윗에 숨겨진 개별 감성을 추출한다. 이후 이들은 규칙기반 통합 방법에 의해 날짜별로 통합되고, 마지막으로 개인의 감성흐름을 보다 직관적으로 관측할 수 있도록 감성흐름그래프로 시각화한다. 제안된 방법의 효용성을 평가하기 위해 두 단계의 실험이 수행되었다. 먼저 4만여 개의 트윗으로부터 제안된 방법의 정확도 평가 실험이 수행되고, 최신 트윗 분석 기술과 비교 분석되었다. 두 번째 실험에서는 40명의 우울증을 가진 사용자와 일반사용자를 구분할 수 있는지에 대한 실험이 수행된 결과, 제안된 기술이 실제 사용자의 감성흐름을 모니터하는데 효율적임을 증명하였다.

Keywords

References

  1. M. Marcus, M. T. Yasamy, M. van Ommeren, D. Chisholm, S. Saxena, "Depression: A global public health concern," WHO Department of Mental Health and Substance Abuse Vol. 1, pp. 6-8, 2012.
  2. Z. Rihmer, "Can better recognition and treatment of depression reduce suicide rates? A brief review," European Psychiatry, Vol. 16, pp. 406-409, 2001. https://doi.org/10.1016/S0924-9338(01)00598-3
  3. L. S. Radloff L S, "The CES-D scale a self-report depression scale for research in the general population," Applied Psychological Measurement, Vol. 1, pp. 385-401, 1977. https://doi.org/10.1177/014662167700100306
  4. G. E. Simon, D. B. Goldberg, G. Tiemens, T. B. Ustun, "Outcomes of recognized and unrecognized depression in an international primary care study," General Hospital Psychiatry, Vol. 21, pp. 97-105, 1999. https://doi.org/10.1016/S0163-8343(98)00072-3
  5. M. De Choudhury, M. Gamon, S. Counts, E. Horvitz, "Predicting depression via social media," ICWSM 2013, pp. 1-10, 2013.
  6. J. Yuan, Q. You, J. Luo, "Sentiment Analysis Using Social Multimedia," In Multimedia Data Mining and Analytics, Springer International Publishing, pp. 31-59, 2015.
  7. S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," ACM CHI, pp. 3187-3196, 2015.
  8. A. Go, L. Huang, R. Bhayani, "Twitter sentiment analysis," Final Projects from CS224N for Spring 2008/2009 at The Stanford Natural Language Processing Group, 2009.
  9. Y. R. Tausczik, J. W. Pennebaker, "The psychological meaning of words: LIWC and computerized text analysis methods," Journal of Language and Social Psychology, Vol. 29, pp. 24-54, 2010. https://doi.org/10.1177/0261927X09351676
  10. M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, A. Kappas, "Sentiment strength detection in short informal text," Journal of the Association for Information Science and Technology, Vol. 61, pp. 2544-2558, 2010.
  11. L. Zhang, R. Ghosh, M. Dekhil, M. Hsu, B. Liu, "Combining Lexicon-based and Learning-based Methods for Twitter Sentiment Analysis," In HP Laboratories Technical Report (89 ed.), 2011.
  12. J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, J. Tang, "Can we understand van Gogh's mood?: Learning to infer affects from images in social networks," MM2012, pp. 857-860, 2012.
  13. A. Mogadala, V. Varma, "Twitter user behavior understanding with mood transition prediction," The 2012 workshop on Data-driven user behavioral modelling and mining from social media, pp. 31-34. 2012.
  14. S. Park, S. W. Lee, J. Kwak, M. Cha, B. Jeong, "Activities on Facebook reveal the depressive state of users," Journal of Medical Internet Research, Vol. 15, 2013.
  15. A. Hogenboom, B. Daniella, F. Flavius, B. Malissa, J. Franciska, K. Uzay, "Exploiting emoticons in sentiment analysis," Annual ACM Symposium on Applied Computing 2013, pp. 703-710. 2013.
  16. S. M. Mohammad, Saif M., and D. T. Peter, "Crowdsourcing a word-emotion association lexicon," Computational Intelligence Vol. 29, pp. 436-465, 2013. https://doi.org/10.1111/j.1467-8640.2012.00460.x
  17. Y. Yang, C. Peng, Z. Wenwu, Y. Shiqiang, "User interest and social influence based emotion prediction for individuals," MM2013, pp. 785-788, 2013.
  18. C. Xu, C. Suleyman, K. Lee, L. Li, "Visual sentiment prediction with deep convolutional neural networks," arXiv preprint arXiv:1411.5731, 2014.
  19. H. Lin, J. Jia, Q. Guo, Y. Xue, Q. Li, J. Huang, "User-level psychological stress detection from social media using deep neural network," MM2014, pp. 507-516, 2014.
  20. Q. You, J. Luo, H. Jin, J. Yang, "Robust image sentiment analysis using progressively trained and domain transferred deep networks," AAAI2015, pp. 381-388, 2015.
  21. S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," CHI2015, pp. 3187-3196, 2015.
  22. D. Borth, C. Tao, J. Rongrong, C. Shih-Fu, "Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content," MM2013, pp. 459-460. 2013.
  23. http://sentistrength.wlv.ac.uk/Sentistrength.
  24. https://en.wikipedia.org/wiki/List_of_emoticons Wikipedia
  25. E.S. Dan-Glauser, K. R. Scherer, "The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance," Behavior research methods, Vol. 43, pp. 468-477, 2011. https://doi.org/10.3758/s13428-011-0064-1
  26. http://photo.net Photo.Net.
  27. S. Kobayashi, COLOR IMAGE SCALE, Kodansha. 1991.
  28. J. Bollen, H. Mao, X. Zeng, "Twitter mood predicts the stock market," Journal of Computational Science, Vol. 2, pp. 1-8, 2010.
  29. S. Zhao, H. Yao, Y. Yang, Y. Zhang, "Affective image retrieval via multi-graph learning," MM2014, pp. 1025-1028. 2014.
  30. G. Katz, N. Ofek, B. Shapira, "ConSent: Context-based sentiment analysis," Knowledge-Based Systems, Vol. 84, pp. 162-178, 2015.
  31. C.K. Chung, J. W. Pennebaker, "Counting little words in Big Data: The psychology of communities, culture, and history," Social Conition and Communication, Psychology Press, pp. 25-42, 2014.
  32. M. De Chounhury, S. Counts, E. Horvitz, "Predicting postpartum changes in emotion and behavior via social media," CHI 2013, pp. 3267-3276, 2013.
  33. H. Sueki, "The association of suicide-related Twitter use with suicidal behaviour: A cross-sectional study of young internet users in Japan," Journal of Affective Disorder, Vol. 170, pp. 155-160, Jan. 2015. https://doi.org/10.1016/j.jad.2014.08.047