Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Recommender system using BERT sentiment analysis

BERT 기반 감성분석을 이용한 추천시스템

  • 박호연 (동국대학교_서울 일반대학원 경영정보학과) ;
  • 김경재 (동국대학교_서울 경영대학 경영정보학과)
  • Received : 2020.12.03
  • Accepted : 2021.03.08
  • Published : 2021.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

If it is difficult for us to make decisions, we ask for advice from friends or people around us. When we decide to buy products online, we read anonymous reviews and buy them. With the advent of the Data-driven era, IT technology's development is spilling out many data from individuals to objects. Companies or individuals have accumulated, processed, and analyzed such a large amount of data that they can now make decisions or execute directly using data that used to depend on experts. Nowadays, the recommender system plays a vital role in determining the user's preferences to purchase goods and uses a recommender system to induce clicks on web services (Facebook, Amazon, Netflix, Youtube). For example, Youtube's recommender system, which is used by 1 billion people worldwide every month, includes videos that users like, "like" and videos they watched. Recommended system research is deeply linked to practical business. Therefore, many researchers are interested in building better solutions. Recommender systems use the information obtained from their users to generate recommendations because the development of the provided recommender systems requires information on items that are likely to be preferred by the user. We began to trust patterns and rules derived from data rather than empirical intuition through the recommender systems. The capacity and development of data have led machine learning to develop deep learning. However, such recommender systems are not all solutions. Proceeding with the recommender systems, there should be no scarcity in all data and a sufficient amount. Also, it requires detailed information about the individual. The recommender systems work correctly when these conditions operate. The recommender systems become a complex problem for both consumers and sellers when the interaction log is insufficient. Because the seller's perspective needs to make recommendations at a personal level to the consumer and receive appropriate recommendations with reliable data from the consumer's perspective. In this paper, to improve the accuracy problem for "appropriate recommendation" to consumers, the recommender systems are proposed in combination with context-based deep learning. This research is to combine user-based data to create hybrid Recommender Systems. The hybrid approach developed is not a collaborative type of Recommender Systems, but a collaborative extension that integrates user data with deep learning. Customer review data were used for the data set. Consumers buy products in online shopping malls and then evaluate product reviews. Rating reviews are based on reviews from buyers who have already purchased, giving users confidence before purchasing the product. However, the recommendation system mainly uses scores or ratings rather than reviews to suggest items purchased by many users. In fact, consumer reviews include product opinions and user sentiment that will be spent on evaluation. By incorporating these parts into the study, this paper aims to improve the recommendation system. This study is an algorithm used when individuals have difficulty in selecting an item. Consumer reviews and record patterns made it possible to rely on recommendations appropriately. The algorithm implements a recommendation system through collaborative filtering. This study's predictive accuracy is measured by Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). Netflix is strategically using the referral system in its programs through competitions that reduce RMSE every year, making fair use of predictive accuracy. Research on hybrid recommender systems combining the NLP approach for personalization recommender systems, deep learning base, etc. has been increasing. Among NLP studies, sentiment analysis began to take shape in the mid-2000s as user review data increased. Sentiment analysis is a text classification task based on machine learning. The machine learning-based sentiment analysis has a disadvantage in that it is difficult to identify the review's information expression because it is challenging to consider the text's characteristics. In this study, we propose a deep learning recommender system that utilizes BERT's sentiment analysis by minimizing the disadvantages of machine learning. This study offers a deep learning recommender system that uses BERT's sentiment analysis by reducing the disadvantages of machine learning. The comparison model was performed through a recommender system based on Naive-CF(collaborative filtering), SVD(singular value decomposition)-CF, MF(matrix factorization)-CF, BPR-MF(Bayesian personalized ranking matrix factorization)-CF, LSTM, CNN-LSTM, GRU(Gated Recurrent Units). As a result of the experiment, the recommender system based on BERT was the best.

추천시스템은 사용자의 기호를 파악하여 물품 구매 결정을 도와주는 역할을 할 뿐만 아니라, 비즈니스 전략의 관점에서도 중요한 역할을 하기에 많은 기업과 기관에서 관심을 갖고 있다. 최근에는 다양한 추천시스템 연구 중에서도 NLP와 딥러닝 등을 결합한 하이브리드 추천시스템 연구가 증가하고 있다. NLP를 이용한 감성분석은 사용자 리뷰 데이터가 증가함에 따라 2000년대 중반부터 활용되기 시작하였지만, 기계학습 기반 텍스트 분류를 통해서는 텍스트의 특성을 완전히 고려하기 어렵기 때문에 리뷰의 정보를 식별하기 어려운 단점을 갖고 있다. 본 연구에서는 기계학습의 단점을 보완하기 위하여 BERT 기반 감성분석을 활용한 추천시스템을 제안하고자 한다. 비교 모형은 Naive-CF(collaborative filtering), SVD(singular value decomposition)-CF, MF(matrix factorization)-CF, BPR-MF(Bayesian personalized ranking matrix factorization)-CF, LSTM, CNN-LSTM, GRU(Gated Recurrent Units)를 기반으로 하는 추천 모형이며, 실제 데이터에 대한 분석 결과, BERT를 기반으로 하는 추천시스템의 성과가 가장 우수한 것으로 나타났다.

Keywords

Acknowledgement

이 논문은 2019년 대한민국 교육부와 한국연구재단의 지원을 받아 수행된 연구임 (NRF-2019S1A5A2A01050194)

References

  1. Akhtar, N., N. Zubair, A. Kumar and T. Ahmad, "Aspect Based Sentiment Oriented Summarization of Hotel Reviews", Procedia computer science, Vol.115, (2017), 563~71. https://doi.org/10.1016/j.procs.2017.09.115
  2. Bahdanau, D., K. Cho and Y. Bengio, "Neural machine translation by jointly learning to align and translate", arXiv preprint arXiv: 1409.0473, (2014)
  3. Bai, P., Y. Xia and Y. Xia "Fusing Knowledge and Aspect Sentiment for Explainable Recommendation", IEEE Access Vol.8, (2020), 137150~137160. https://doi.org/10.1109/ACCESS.2020.3012347
  4. Chelliah, M. and S. Sarkar, Product Recommendations Enhanced with Reviews. Proceedings of the Eleventh ACM Conference on Recommender Systems, 2017.
  5. Chen, R.-C., "User Rating Classification Via Deep Belief Network Learning and Sentiment Analysis", IEEE Transactions on Computational Social Systems, Vol.6, No.3, (2019), 535~46. https://doi.org/10.1109/tcss.2019.2915543
  6. D'Addio, R. M., M. A. Domingues and M. G. Manzato, "Exploiting Feature Extraction Techniques on Users' Reviews for Movies Recommendation", Journal of the Brazilian Computer Society, Vol.23, No.1, (2017), 7. https://doi.org/10.1186/s13173-017-0057-8
  7. Da'u, A. and N. Salim, "Sentiment-aware deep recommender system with neural attention networks", IEEE Access, Vol.7(2019), 45472~45484. https://doi.org/10.1109/ACCESS.2019.2907729
  8. Devlin, J., M.-W. Chang, K. Lee and K. Toutanova, "Bert: pre-training of Deep Bidirectional Transformers for Language Understanding", arXiv preprint arXiv:1810.04805, (2018)
  9. Du, Y. L., X. W. Meng and Y. J. Zhang, "Cvtm: A Content-Venue-Aware Topic Model for Group Event Recommendation", IEEE Transactions on Knowledge and Data Engineering, Vol.32, No.7, (Jul 1 2020), 1290~1303. ://WOS:000543006000005 https://doi.org/10.1109/tkde.2019.2904066
  10. Garcia-Cumbreras, M. A., A. Montejo-Raez and M. C. Diaz-Galiano, "Pessimists and Optimists: Improving Collaborative Filtering through Sentiment Analysis", Expert Systems with Applications, Vol.40, No.17, (2013), 6758~6765. https://doi.org/10.1016/j.eswa.2013.06.049
  11. Hu, S., A. Kumar, F. Al-Turjman, S. Gupta and S. Seth, "Reviewer Credibility and Sentiment Analysis Based User Profile Modelling for Online Product Recommendation", IEEE Access, Vol.8, (2020), 26172~26189. https://doi.org/10.1109/ACCESS.2020.2971087
  12. Hyun, J., S. Ryu and S.-Y. T. Lee "How to Improve the Accuracy of Recommendation Systems: Combining Ratings and Review Texts Sentiment Scores", Journal of Intelligence and Information Systems, Vol.25, No.1, (2019), 219~239. https://doi.org/10.13088/JIIS.2019.25.1.219
  13. Jawaheer, G., M. Szomszor and P. Kostkova, Comparison of Implicit and Explicit Feedback from an Online Music Recommendation Service. proceedings of the 1st international workshop on information heterogeneity and fusion in recommender systems,, (2010).
  14. Kim, K.-W. and D.-H. Park, "Individual Thinking Style Leads Its Emotional Perception: Development of Web-Style Design Evaluation Model and Recommendation Algorithm Depending on Consumer Regulatory Focus", Journal of Intelligence and Information Systems, Vol.24, No.4, (2018), 171~196. https://doi.org/10.13088/JIIS.2018.24.4.171
  15. Ku, M. J. and H. Ahn "A Hybrid Recommender System Based on Collaborative Filtering with Selective Use of Overall and Multicriteria Ratings", Journal of Intelligence and Information Systems, Vol.24, No.2, (2018), 85~109. https://doi.org/10.13088/JIIS.2018.24.2.085
  16. Kumar, S., K. De and P. P. Roy "Movie Recommendation System Using Sentiment Analysis from Microblogging Data", IEEE Transactions on Computational Social Systems, (2020)
  17. Majumder, N., S. Poria, H. Peng, N. Chhaya, E. Cambria and A. Gelbukh, "Sentiment and Sarcasm Classification with Multitask Learning", IEEE Intelligent Systems, Vol.34, No.3, (2019), 38~43. https://doi.org/10.1109/mis.2019.2904691
  18. Otsuka, E., S. A. Wallace and D., Design and Evaluation of a Twitter Hashtag Recommendation System. Proceedings of the 18th International Database Engineering & Applications Symposium, 2014.
  19. Qian, X. M., H. Feng, G. S. Zhao and T. Mei, "Personalized Recommendation Combining User Interest and Social Circle", IEEE Transactions on Knowledge and Data Engineering, Vol.26, No.7, (Jul 2014), 1763~77. ://WOS:000340205700017. https://doi.org/10.1109/tkde.2013.168
  20. Rothe, S., S. Narayan and A. Severyn, "Leveraging pre-trained Checkpoints for Sequence Generation Tasks", Transactions of the Association for Computational Linguistics, Vol.8, (2020), 264~280. https://doi.org/10.1162/tacl_a_00313
  21. Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, Attention Is All You Need. Advances in neural information processing systems, 2017
  22. Xie, Q., X. Zhang, Y. Ding and M. Song, "Monolingual and Multilingual Topic Analysis Using Lda and Bert Embeddings", Journal of Informetrics, Vol.14, No.3, (2020), 101055. https://doi.org/10.1016/j.joi.2020.101055
  23. Xie, R., C. Ling, Y. Wang, R. Wang, F. Xia and L. Lin "Deep Feedback Network for Recommendation", Proceedings of IJCAI-PRICAI, (2020).
  24. Yin, H., W. Wang, L. Chen, X. Du, Q. V. H. Nguyen and Z. Huang, "Mobi-Sage-Rs: A Sparse Additive Generative Model-Based Mobile Application Recommender System", Knowledge-Based Systems, Vol.157, (2018), 68~80. https://doi.org/10.1016/j.knosys.2018.05.028
  25. Zangerle, E., W. Gassler and G. Specht, "On the Impact of Text Similarity Functions on Hashtag Recommendations in Microblogging Environments", Social network analysis and mining , Vol.3, No.4, (2013), 889~898. https://doi.org/10.1007/s13278-013-0108-x
  26. Zeng, Z., C. Xiao, Y. Yao, R. Xie, Z. Liu, F. Lin, L. Lin and M. Sun, "Knowledge Transfer Via pre-training for Recommendation: A Review and Prospect", arXiv preprint arXiv:2009.09226, (2020)
  27. Zhang, W. and J. Wang, "Integrating Topic and Latent Factors for Scalable Personalized Review-Based Rating Prediction", IEEE Transactions on Knowledge and Data Engineering, Vol.28, No.11, (2016), 3013~3027. https://doi.org/10.1109/TKDE.2016.2598740
  28. Zhao, F., Y. Zhu, H. Jin and L. T. Yang, "A Personalized Hashtag Recommendation Approach Using Lda-Based Topic Model in Microblog Environment", Future Generation Computer Systems, Vol.65, (2016), 196~206. https://doi.org/10.1016/j.future.2015.10.012
  29. Zhou, Y., X. Wang, M. Zhang, J. Zhu, R. Zheng and Q. Wu, "Mpce: A Maximum Probability Based Cross Entropy Loss Function for Neural Network Classification", IEEE Access, Vol.7, (2019), 146331~146341. https://doi.org/10.1109/ACCESS.2019.2946264