Journal of Internet of Things and Convergence (사물인터넷융복합논문지)

The Korea Internet of Things Society (한국사물인터넷학회)
권호 표지
DOI QR코드

DOI QR Code

Information-providing Application Based on Web Crawling

웹 크롤링을 통한 개인 맞춤형 정보제공 애플리케이션

  • Ju-Hyeon Kim (School of Computer and Artificial Intelligence Engineering, Pukyong National University ) ;
  • Jeong-Eun Choi (School of Computer and Artificial Intelligence Engineering, Pukyong National University) ;
  • U-Gyeong Shin (School of Computer and Artificial Intelligence Engineering, Pukyong National University ) ;
  • Min-Jun Piao (School of Computer and Artificial Intelligence Engineering, Pukyong National University) ;
  • Tae-Kook Kim (School of Computer and Artificial Intelligence Engineering, Pukyong National University )
  • 김주현 (국립부경대학교 컴퓨터.인공지능공학부 ) ;
  • 최정은 (국립부경대학교 컴퓨터.인공지능공학부 ) ;
  • 신우경 (국립부경대학교 컴퓨터.인공지능공학부 ) ;
  • 박민준 (국립부경대학교 컴퓨터.인공지능공학부 ) ;
  • 김태국 (국립부경대학교 컴퓨터.인공지능공학부)
  • Received : 2023.12.29
  • Accepted : 2024.02.04
  • Published : 2024.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

This paper presents the implementation of a personalized real-time information-providing application utilizing filtering and web crawling technologies. The implemented application performs web crawling based on the user-set keywords within web pages, using the Jsoup library as a basis for the selected keywords. The crawled data is then stored in a MySQL database. The stored data is presented to the user through an application implemented using Flutter. Additionally, mobile push notifications are provided using Firebase Cloud Messaging (FCM). Through these methods, users can efficiently obtain the desired information quickly. Furthermore, there is an expectation that this approach can be applied to the Internet of Things (IoT) where big data is generated, allowing users to receive only the information they need.

본 논문에서는 필터링(Filtering)과 웹 크롤링(Web Crawling) 기술을 이용하여 개인 맞춤형 실시간 정보제공 애플리케이션을 구현하였다. 구현한 애플리케이션은 사용자가 설정한 키워드를 웹페이지 내에서 사용자가 선택한 키워드를 기준으로 Jsoup 라이브러리를 통해 웹 크롤링을 수행하고, MySQL 데이터베이스에 저장한다. 저장한 데이터는 Flutter를 이용해 구현한 애플리케이션으로 사용자에게 제공한다. 또한 FCM(Firebase Cloud Messaging)을 이용하여 모바일 푸시 알람을 제공한다. 이를 통해 사용자는 원하는 정보를 빠르고 효율적으로 얻을 수 있다. 또한 빅데이터가 생성되는 사물인터넷(Internet of things)에도 적용하여 사용자에게 필요한 정보만 제공할 수 있을 것으로 기대한다.

Keywords

Acknowledgement

이 성과는 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(RS-2023-00242528).

References

  1. C.W.Na, B.W.On, "A proposal on a proactive crawling approach with analysis of state-of-the-art web crawling algorithms," Korean Society for Internet Information, Vol.20, No.3, pp.43-59, 2019. 
  2. J.H.Kim, E.J.Kim, "WCTT: Web Crawling System based on HTML Document Formalization," The Korea Institute of Information and Commucation Engineering, Vol.26, No.4, pp.495-502, 2021. 
  3. Y.J.Lee, "Estimation of maximum object size satisfying mean response time constraint in web service environment," Journal of Internet of Things and Convergence, Vol.9, No.3, pp.1-6, 2023.  https://doi.org/10.20465/KIOTS.2023.9.3.001
  4. B.H.Lee, "HTML specification and semantics analysis of korean news sites," Journal of Digital Contents Society, Vol.18, No.5, pp.949-956, 2017.  https://doi.org/10.9728/DCS.2017.18.5.949
  5. H.I.Lee, J.H.Cha, "A Study on the Recognition of Korea Armed Forces Nursing Academy and Military Nursing Officers through Web Crawling and Text Mining," Journal of the Korea Academia-Industrial cooperation Society, Vol.24, No.5, pp.381-388, 2023.  https://doi.org/10.5762/KAIS.2023.24.5.381
  6. Firebase, Firebase Cloud Messaging[Internet], https://firebase.google.com/docs/cloud-messaging. 
  7. A.S.Oh, "Design and Implementation of Platform for Monitoring of Notification System in Firebase Message," Journal of Information & Communication Convergence Engineering, Vol.19, No.1, pp.16-21, 2021. 
  8. S.Kwak, "Product Recommendation System Based on User Purchase Criteria and Product Reviews," Paichai university, Master's thesis, 2021. 
  9. J.K.Kim, K.H.Sim, Y.S.Lee, Y.H.Lim, "Development of real-time monitoring web BBS and the alerts service using mobile web," Journal of Digital Contents Society, Vol.13, No.1, pp.1-11, 2012.  https://doi.org/10.9728/DCS.2012.13.1.001
  10. Y.H.Lee, "Implementation Of Web Crawling Program For Efficient Use Of Big Data," Jounal of The Korea Society of Information Technology Policy & Management, Vol.12, No.5, pp.1983-1989, 2020. 
  11. H.W.Park, J.H.Choi, S.G.Hwang, M.S.Park, S.U.Noh,, K.H.Chung, K.H.Choi, "Development of verification tool for the soundness of software requirements specification using clustering techniques," The Journal of Korean Institute of Next Generation Computing, Vol.17, No6, pp.54-65., 2021.  https://doi.org/10.23019/KINGPC.17.6.202112.005
  12. Y.K.Kim, "A Study on the Completeness Measurement of Requirements Specifications for Software Completion Appraisal," Journal of Software Assessment and Valuation, Vol.19, No.1, pp.11-18, 2023.  https://doi.org/10.29056/jsav.2023.3.02
  13. T.K.Kim, "Spatial Crowdedness Measurement System using IoT and Amazon Web Services," Journal of Internet of Things and Convergence, Vol.9, No.4, pp.15-20, 2023.  https://doi.org/10.20465/KIOTS.2023.9.4.015
  14. Amazon, Amazon Web Services[Internet], https://aws.amazon.com. 
  15. Flutter, Flutter - Build apps for any screen[Internet], https://www.figma.com. 
  16. Figma, Figma: The Collaborative Interface Design Tool[Internet], https://flutter.dev. 
  17. Kakao, Kakao developers[Internet], https://developers.kakao.com.