Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Item Recommendation Technique Using Spark

Spark를 이용한 항목 추천 기법에 관한 연구

  • Yun, So-Young (Department of Computer Engineering, Pukyong National University) ;
  • Youn, Sung-Dae (Department of Computer Engineering, Pukyong National University)
  • Received : 2018.03.28
  • Accepted : 2018.05.07
  • Published : 2018.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

With the spread of mobile devices, the users of social network services or e-commerce sites have increased dramatically, and the amount of data produced by the users has increased exponentially. E-commerce companies have faced a task regarding how to extract useful information from a vast amount of data produced by the users. To solve this problem, there are various studies applying big data processing technique. In this paper, we propose a collaborative filtering method that applies the tag weight in the Apache Spark platform. In order to elevate the accuracy of recommendation, the proposed method refines the tag data in the preprocessing process and categorizes the items and then applies the information of periods and tag weight to the estimate rating of the items. After generating RDD, we calculate item similarity and prediction values and recommend items to users. The experiment result indicated that the proposed method process large amounts of data quickly and improve the appropriateness of recommendation better.

모바일 기기의 확산으로 소셜 네트워크 서비스나 전자상거래 사이트의 사용자 수가 급증하고 있고 사용자들이 남긴 데이터의 양도 기하급수적으로 증가하고 있다. 그로 인해 전자 상거래 기업들은 사용자들이 남긴 방대한 양의 데이터로부터 어떻게 유용한 정보를 추출할 것인가 하는 과제를 갖게 되었다. 이러한 문제를 해결하기 위해 추천 시스템에 빅 데이터 처리 기법을 적용한 다양한 연구들이 이루어지고 있다. 본 논문에서는 Apache Spark 플랫폼에서 Tag 가중치를 적용한 협업 필터링 기법을 사용한 추천방식을 제안한다. 제안하는 기법은 추천의 정확성을 높이기 위해 전처리 과정에서 Tag 데이터를 정제하고 아이템을 분류한 후 아이템 평가값에 기간 정보와 Tag 가중치를 적용하여 사용한다. RDD(Resilient Distributed Dataset)를 생성한 후 아이템 유사도와 예측값을 구하고 사용자에게 아이템을 추천한다. 실험을 통해 제안 하는 기법이 대량의 데이터를 빠르게 처리하고 추천의 적합성도 향상되는 것을 확인하였다.

Keywords

References

  1. P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl, "Grouplens: an open architecture for collaborative filtering of Netnews," Proceedings of the ACM Conference on Computer Supported Cooperative Work, pp. 175-186, New York, NY, USA, Oct. 1994.
  2. G. Linden, B. Smith, and J. York, "Amazon.com recommendations: item-to-item collaborative filtering," IEEE Internet Computing, vol. 7, no. 1, pp. 76-80, January/February 2003. https://doi.org/10.1109/MIC.2003.1167344
  3. S. Meng, W. Dou, X. Zhang, and J. Chen, "KASR : A Keyword-Aware Service Recommendation Method on MapReduce for Big Data Applications," IEEE Transactions on Parallel and Distributed Systems, vol. 25, no.12, pp.3221-3231, Dec. 2014. https://doi.org/10.1109/TPDS.2013.2297117
  4. Y. Shang, Z. Li, W. Qu, Y. Xu, Z. Song, and X. Zhou, "Scalable Collaborative Filtering Recommendation Algorithm with MapReduce," 2014 IEEE 12th International Conference on Dependable, Autonomic and Secure Computing, pp.103-108, Aug. 2014
  5. P. Ghuli, A. Ghosh, and R. Shettar, "A collaborative filtering recommendation engine in a distributed environment," 2014 International Conference on Contemporary Computing and Informatics(IC3I), pp. 568-574, Nov. 2014.
  6. B.Kupisz, O. Unold, "Collaborative Filtering Recommen- dation Algorithm based on Hadoop and Spark", 2015 IEEE International Conference on Industriial Technology(ICIT), pp.1510-1513, Mar. 2015.
  7. C. Sardianos, I. Varlamis, and M. Eirinaki," Scaling Collaborative Filtering to large-scale Bipartite Rating Graphs," 2017 IEEE Third International Conference on Big Data Computing Service and Applications, pp.70-79, Apr. 2017.
  8. S. Yun, S. Youn, "Recommendation System Using Big Data Processing Technique," Journal of Korea Institute of Information and Communication Engineering, Vol. 21, No.6, pp.1183-1190 , Jun. 2017. https://doi.org/10.6109/JKIICE.2017.21.6.1183
  9. J.S.Breese, D.Heckerman, and C. Kadle, "Empirical Analysis of Predictive Algorithms for Collaborative Filtering," in Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, pp.43-52, Jul. 1998.
  10. B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, "Item-based Collaborative Filtering Recommendation Algorithms," in Proceedings of the 10th International World Wide Web Conference, ACM Press, pp. 285-295, 2001.
  11. O.J. Lee, E.S. You, "Predictive Clustering-based Collaborative Filtering Technique for Performance-Stability of Recommendation System", Journal of Intelligence and Information Systems, vol.21, no. 1 , pp.119-142, Mar. 2015. https://doi.org/10.13088/jiis.2015.21.1.119
  12. S.K. Gorakala, Building Recommendation Engines, Birmingham, Packt Publishing, 2016.
  13. S. Yang, B. Wu, "Large Scale Video Data Analysis Based on Spark," 2015 International Conference on Cloud Computing and Big Data(CCBD), pp.209-212, Nov. 2015.
  14. M. Zaharia, H. Karau, A. Konwinski, and P. Wendell Learning Spark, Sebastopol. CA: O'REILLY, 2015.
  15. U. Ramachandran, H. Venkateswaran, A. Sivasubramaniam, and A. Singla, "Issues in understanding the scalability of parallel systems," in Proceedings of the First International Workshop on Parallel Processing, pp.399-404. Dec. 1994.
  16. S.H.H. Ding, B.C.M. Fung and P. Charland, "Kam1n0: MapReduce-based Assembly Clone Search for Reverse Engineering," KDD'16 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 461-470, Aug. 2016.
  17. A. Stanescu, S. Nagar and D. Caragea "A Hybrid Recommender System: User Profiling from Keywords and Ratings," WI-IAT '13 Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence(WI) and Intelligent Agent Technologies(IAT), vol. 01, pp. 73-80, Nov. 2013.