Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Feature Analysis for Detecting Mobile Application Review Generated by AI-Based Language Model

  • Received : 2020.09.15
  • Accepted : 2021.02.03
  • Published : 2022.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Mobile applications can be easily downloaded and installed via markets. However, malware and malicious applications containing unwanted advertisements exist in these application markets. Therefore, smartphone users install applications with reference to the application review to avoid such malicious applications. An application review typically comprises contents for evaluation; however, a false review with a specific purpose can be included. Such false reviews are known as fake reviews, and they can be generated using artificial intelligence (AI)-based text-generating models. Recently, AI-based text-generating models have been developed rapidly and demonstrate high-quality generated texts. Herein, we analyze the features of fake reviews generated from Generative Pre-Training-2 (GPT-2), an AI-based text-generating model and create a model to detect those fake reviews. First, we collect a real human-written application review from Kaggle. Subsequently, we identify features of the fake review using natural language processing and statistical analysis. Next, we generate fake review detection models using five types of machine-learning models trained using identified features. In terms of the performances of the fake review detection models, we achieved average F1-scores of 0.738, 0.723, and 0.730 for the fake review, real review, and overall classifications, respectively.

Keywords

Acknowledgement

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. NRF-2020R1I1A3073313).

References

  1. Y. Liu, Z. Bao, Z. Zhang, D. Tang, and F. Xiong, "Information cascades prediction with attention neural network," Human-centric Computing and Information Sciences, vol. 10, article no, 13, 2020. https://doi.org/10.1186/s13673-020-00218-w
  2. Z. Zhang, J. Jing, X. Wang, K. K. R. Choo, and B. B. Gupta, "A crowdsourcing method for online social networks security assessment based on human-centric computing," Human-centric Computing and Information Sciences, vol. 10, article no, 23, 2020. https://doi.org/10.1186/s13673-020-00230-0
  3. M. Harman, Y. Jia, and Y. Zhang, "App store mining and analysis: MSR for app stores," in Proceedings of 2012 9th IEEE Working Conference on Mining Software Repositories (MSR), Zurich, Switzerland, 2012, pp. 108-111.
  4. M. Talal, A. A. Zaidan, B. B. Zaidan, O. S. Albahri, M. A. Alsalem, A. S. Albahri, et al., "Comprehensive review and analysis of anti-malware apps for smartphones," Telecommunication Systems, vol. 72, no, 2, pp. 285-337, 2019. https://doi.org/10.1007/s11235-019-00575-7
  5. S. Y. Choi, C. G. Lim, and Y. M. Kim, "Automated link tracing for classification of malicious websites in malware distribution networks," Journal of Information Processing Systems, vol. 15, no, 1, pp. 100-115, 2019. https://doi.org/10.3745/JIPS.03.0107
  6. H. Chen, D. He, S. Zhu, and J. Yang, "Toward detecting collusive ranking manipulation attackers in mobile app markets," in Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, Abu Dhabi, United Arab Emirates, 2017, pp. 58-70.
  7. N. Genc-Nayebi and A. Abran, "A systematic literature review: opinion mining studies from mobile app store user review," Journal of Systems and Software, vol. 125, pp. 207-219, 2017. https://doi.org/10.1016/j.jss.2016.11.027
  8. D. He, M. Pan, K. Hong, Y. Cheng, S. Chan, X. Liu, and N. Guizani, "Fake review detection based on PU learning and behavior density," IEEE Network, vol. 34, no. 4, pp. 298-303, 2020. https://doi.org/10.1109/mnet.001.1900542
  9. Y. S. Jeong and J. H. Park, "Learning algorithms in AI system and services," Journal of Information Processing System, vol. 15, no, 5, pp. 1029-1035, 2019. https://doi.org/10.3745/jips.02.0118
  10. A. See, A. Pappu, R. Saxena, A. Yerukola, and C. D. Manning, "Do massively pretrained language models make better storytellers?," 2019 [Online]. Available: https://arxiv.org/abs/1909.10705.
  11. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," 2019 [Online]. Available: https://d4mucfpksywv.cloudfront.net/better-languagemodels/language-models.pdf.
  12. Y. Jang, C. H. Park, and Y. S. Seo, "Fake news analysis modeling using quote retweet," Electronics, vol. 8, no, 12, article no. 1377, 2019. https://doi.org/10.3390/electronics8121377
  13. M. Ott, Y. Choi, C. Cardie, and J. T. Hancock, "Finding deceptive opinion spam by any stretch of the imagination," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, 2011, pp. 309-319.
  14. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Improving language understanding by generative pre-training," 2018 [Online]. Available: https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf.
  15. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, et al., "Language models are few-shot learners," Advances in Neural Information Processing Systems, vol. 33, pp. 1877-1901, 2020.
  16. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding," 2019 [Online]. Available: https://arxiv.org/abs/1810.04805.
  17. S. Gehrmann, H. Strobelt, and A. M. Rush, "GLTR: statistical detection and visualization of generated text," [2019]. Available: https://arxiv.org/abs/1906.04043.
  18. L. Zhang, X. Y. Huang, J. Jiang, and Y. K. Hu, "CSLabel: an approach for labelling mobile app reviews," Journal of Computer Science and Technology, vol. 32, no. 6, pp. 1076-1089, 2017. https://doi.org/10.1007/s11390-017-1784-1
  19. D. Martens and W. Maalej, "Towards understanding and detecting fake reviews in app stores," Empirical Software Engineering, vol. 24, no. 6, pp. 3316-3355, 2019. https://doi.org/10.1007/s10664-019-09706-9
  20. K. Ouazzane, J. Li, H. B. Jun, Y. Jing, and R. Boyd, "An artificial Intelligence-based language modeling framework," Expert Systems with Applications, vol. 39, no, 5, pp. 5960-5970, 2012. https://doi.org/10.1016/j.eswa.2011.11.121
  21. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi, "Defending against neural fake news," Advances in Neural Information Processing Systems, vol. 32, pp. 9054-9065, 2019.
  22. V. Kieuvongngam, B. Tan, and Y. Niu, "Automatic text summarization of COVID-19 medical research articles using BERT and GPT-2," 2020 [Online]. Available: https://arxiv.org/abs/2006.01997.
  23. S. Barrio, "Writing the next American hit: using GPT-2 to explore the possibility of creating successful AIgenerated song lyrics," 2020 [Online]. Available: https://digital.kenyon.edu/cgi/viewcontent.cgi?article=1011&context=dh_iphs_prog.
  24. D. I. Adelani, H. Mai, F. Fang, H. H. Nguyen, J. Yamagishi, and I. Echizen, "Generating sentiment-preserving fake online reviews using machine language models and their human- and machine-based detection," in Advanced Information Networking and Applications. Cham, Switzerland: Springer, 2020, pp. 1341-1354.
  25. Y. Nishi, A. Suge, and H. Takahashi, "Construction of news article evaluation system using language generation model," in Agents and Multi-Agent Systems: Technologies and Applications 2020. Singapore: Springer, 2020, pp. 313-320.
  26. S. Kreps, R. M. McCain, and M. Brundage, "All the news that's fit to fabricate: AI-generated text as a tool of media misinformation," Journal of Experimental Political Science, vol. 9, no. 1, pp. 104-117, 2022. https://doi.org/10.1017/XPS.2020.37
  27. A. Destine-DeFreece, S. Handelsman, T. Light Rake, A. Merkel, and G. Moses, "Can GPT-2 replace a Sex and the City writers' room?," 2019 [Online]. Available: https://digital.kenyon.edu/dh_iphs_ai/15/.
  28. Y. Liao, Y. Wang, Q. Liu, and X. Jiang, "GPT-based generation for classical Chinese," 2019 [Online]. Available: https://arxiv.org/abs/1907.00151.
  29. T. Fagni, F. Falchi, M. Gambini, A. Martella, and M. Tesconi, "TweepFake: about detecting deepfake tweets," 2021 [Online]. Available: https://arxiv.org/abs/2008.00036.
  30. Z. Horvitz, N. Do, and M. L. Littman, "Context-driven satirical headline generation," in Proceedings of the 2nd Workshop on Figurative Language Processing, Virtual Event, 2020, pp. 40-50.
  31. J. S. Lee and J. Hsiang, "Patent classification by fine-tuning BERT language," World Patent Information, vol. 61, article no. 101965, 2020. https://doi.org/10.1016/j.wpi.2020.101965
  32. W. Huang, X. Liao, Z. Xie, J. Qian, B. Zhuang, S. Wang, and J. Xiao, "Generating reasonable legal text through the combination of language modeling and question answering," in Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI), Virtual Event, 2020, pp. 3687-3693.
  33. J. Peng, P. Ni, J. Zhu, Z. Dai, Y. Li, G. Li, and X. Bai, "Automatic generation of electronic medical record based on GPT2 model," in Proceedings of 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, 2020, pp. 6180-6182.
  34. J. Salminen, M. Hopf, S. A. Chowdhury, S. G. Jung, H. Almerekhi, and B. J. Jansen, "Developing an online hate classifier for multiple social media platforms," Human-centric Computing and Information Sciences, vol. 10, article no. 1, 2020. https://doi.org/10.1186/s13673-019-0205-6