Monitoring Mood Trends of Twitter Users using Multi-modal Analysis method of Texts and Images

텍스트 및 영상의 멀티모달분석을 이용한 트위터 사용자의 감성 흐름 모니터링 기술

  • Received : 2017.11.13
  • Accepted : 2018.01.20
  • Published : 2018.01.28


In this paper, we propose a novel method for monitoring mood trend of Twitter users by analyzing their daily tweets for a long period. Then, to more accurately understand their tweets, we analyze all types of content in tweets, i.e., texts and emoticons, and images, thus develop a multimodal sentiment analysis method. In the proposed method, two single-modal analyses first are performed to extract the users' moods hidden in texts and images: a lexicon-based and learning-based text classifier and a learning-based image classifier. Thereafter, the extracted moods from the respective analyses are combined into a tweet mood and aggregated a daily mood. As a result, the proposed method generates a user daily mood flow graph, which allows us for monitoring the mood trend of users more intuitively. For evaluation, we perform two sets of experiment. First, we collect the data sets of 40,447 data. We evaluate our method via comparing the state-of-the-art techniques. In our experiments, we demonstrate that the proposed multimodal analysis method outperforms other baselines and our own methods using text-based tweets or images only. Furthermore, to evaluate the potential of the proposed method in monitoring users' mood trend, we tested the proposed method with 40 depressive users and 40 normal users. It proves that the proposed method can be effectively used in finding depressed users.


Convergence;Twitter user recognition;multi-modal analysis;mood trend monitoring;support vector machine;context recognition


  1. M. Marcus, M. T. Yasamy, M. van Ommeren, D. Chisholm, S. Saxena, "Depression: A global public health concern," WHO Department of Mental Health and Substance Abuse Vol. 1, pp. 6-8, 2012.
  2. Z. Rihmer, "Can better recognition and treatment of depression reduce suicide rates? A brief review," European Psychiatry, Vol. 16, pp. 406-409, 2001.
  3. L. S. Radloff L S, "The CES-D scale a self-report depression scale for research in the general population," Applied Psychological Measurement, Vol. 1, pp. 385-401, 1977.
  4. G. E. Simon, D. B. Goldberg, G. Tiemens, T. B. Ustun, "Outcomes of recognized and unrecognized depression in an international primary care study," General Hospital Psychiatry, Vol. 21, pp. 97-105, 1999.
  5. M. De Choudhury, M. Gamon, S. Counts, E. Horvitz, "Predicting depression via social media," ICWSM 2013, pp. 1-10, 2013.
  6. J. Yuan, Q. You, J. Luo, "Sentiment Analysis Using Social Multimedia," In Multimedia Data Mining and Analytics, Springer International Publishing, pp. 31-59, 2015.
  7. S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," ACM CHI, pp. 3187-3196, 2015.
  8. A. Go, L. Huang, R. Bhayani, "Twitter sentiment analysis," Final Projects from CS224N for Spring 2008/2009 at The Stanford Natural Language Processing Group, 2009.
  9. Y. R. Tausczik, J. W. Pennebaker, "The psychological meaning of words: LIWC and computerized text analysis methods," Journal of Language and Social Psychology, Vol. 29, pp. 24-54, 2010.
  10. M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, A. Kappas, "Sentiment strength detection in short informal text," Journal of the Association for Information Science and Technology, Vol. 61, pp. 2544-2558, 2010.
  11. L. Zhang, R. Ghosh, M. Dekhil, M. Hsu, B. Liu, "Combining Lexicon-based and Learning-based Methods for Twitter Sentiment Analysis," In HP Laboratories Technical Report (89 ed.), 2011.
  12. J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, J. Tang, "Can we understand van Gogh's mood?: Learning to infer affects from images in social networks," MM2012, pp. 857-860, 2012.
  13. A. Mogadala, V. Varma, "Twitter user behavior understanding with mood transition prediction," The 2012 workshop on Data-driven user behavioral modelling and mining from social media, pp. 31-34. 2012.
  14. S. Park, S. W. Lee, J. Kwak, M. Cha, B. Jeong, "Activities on Facebook reveal the depressive state of users," Journal of Medical Internet Research, Vol. 15, 2013.
  15. A. Hogenboom, B. Daniella, F. Flavius, B. Malissa, J. Franciska, K. Uzay, "Exploiting emoticons in sentiment analysis," Annual ACM Symposium on Applied Computing 2013, pp. 703-710. 2013.
  16. S. M. Mohammad, Saif M., and D. T. Peter, "Crowdsourcing a word-emotion association lexicon," Computational Intelligence Vol. 29, pp. 436-465, 2013.
  17. Y. Yang, C. Peng, Z. Wenwu, Y. Shiqiang, "User interest and social influence based emotion prediction for individuals," MM2013, pp. 785-788, 2013.
  18. C. Xu, C. Suleyman, K. Lee, L. Li, "Visual sentiment prediction with deep convolutional neural networks," arXiv preprint arXiv:1411.5731, 2014.
  19. H. Lin, J. Jia, Q. Guo, Y. Xue, Q. Li, J. Huang, "User-level psychological stress detection from social media using deep neural network," MM2014, pp. 507-516, 2014.
  20. Q. You, J. Luo, H. Jin, J. Yang, "Robust image sentiment analysis using progressively trained and domain transferred deep networks," AAAI2015, pp. 381-388, 2015.
  21. S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," CHI2015, pp. 3187-3196, 2015.
  22. D. Borth, C. Tao, J. Rongrong, C. Shih-Fu, "Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content," MM2013, pp. 459-460. 2013.
  24. Wikipedia
  25. E.S. Dan-Glauser, K. R. Scherer, "The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance," Behavior research methods, Vol. 43, pp. 468-477, 2011.
  26. Photo.Net.
  27. S. Kobayashi, COLOR IMAGE SCALE, Kodansha. 1991.
  28. J. Bollen, H. Mao, X. Zeng, "Twitter mood predicts the stock market," Journal of Computational Science, Vol. 2, pp. 1-8, 2010.
  29. S. Zhao, H. Yao, Y. Yang, Y. Zhang, "Affective image retrieval via multi-graph learning," MM2014, pp. 1025-1028. 2014.
  30. G. Katz, N. Ofek, B. Shapira, "ConSent: Context-based sentiment analysis," Knowledge-Based Systems, Vol. 84, pp. 162-178, 2015.
  31. C.K. Chung, J. W. Pennebaker, "Counting little words in Big Data: The psychology of communities, culture, and history," Social Conition and Communication, Psychology Press, pp. 25-42, 2014.
  32. M. De Chounhury, S. Counts, E. Horvitz, "Predicting postpartum changes in emotion and behavior via social media," CHI 2013, pp. 3267-3276, 2013.
  33. H. Sueki, "The association of suicide-related Twitter use with suicidal behaviour: A cross-sectional study of young internet users in Japan," Journal of Affective Disorder, Vol. 170, pp. 155-160, Jan. 2015.