Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

HTML Text Extraction Using Tag Path and Text Appearance Frequency

태그 경로 및 텍스트 출현 빈도를 이용한 HTML 본문 추출

  • Kim, Jin-Hwan (Department of Computer Science & Engineering, Korea University of Technology and Education) ;
  • Kim, Eun-Gyung (School of Computer Science & Engineering, Korea University of Technology and Education)
  • Received : 2021.09.09
  • Accepted : 2021.09.27
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In order to accurately extract the necessary text from the web page, the method of specifying the tag and style attributes where the main contents exist to the web crawler has a problem in that the logic for extracting the main contents. This method needs to be modified whenever the web page configuration is changed. In order to solve this problem, the method of extracting the text by analyzing the frequency of appearance of the text proposed in the previous study had a limitation in that the performance deviation was large depending on the collection channel of the web page. Therefore, in this paper, we proposed a method of extracting texts with high accuracy from various collection channels by analyzing not only the frequency of appearance of text but also parent tag paths of text nodes extracted from the DOM tree of web pages.

웹 페이지에서 필요한 텍스트를 정확하게 추출하기 위해 본문이 존재하는 곳의 태그와 스타일 속성을 웹 크롤러에 명시하는 방법은 웹 페이지 구성이 변경될 때마다 본문을 추출하는 로직을 수정해야 하는 문제가 있다. 이러한 문제점을 해결하기 위해 이전 연구에서 제안한 텍스트의 출현 빈도를 분석하여 본문을 추출하는 방법은 웹 페이지의 수집 채널에 따라 성능 편차가 크다는 한계점이 있었다. 따라서 본 논문에서는 텍스트의 출현 빈도뿐만 아니라 웹 페이지의 DOM 트리로부터 추출된 텍스트 노드의 부모 태그 경로를 분석하여 다양한 수집 채널에서 높은 정확도로 본문을 추출하는 방법을 제안하였다.

Keywords

References

  1. Y. J. Kim, H. S. Kim, and H. S. Kim, "Understanding the Effects of COVID-19 on the Starbucks Perception through Big Data Analytics: A Comparative Study," Culinary Science & Hospitality Research, vol. 27, no. 6, pp. 276-279, 2021.
  2. Y. R. Suh, K. P. Koh, and J. W. Lee, "An analysis of the change in media's reports and attitudes about face masks during the COVID-19 pandemic in South Korea: a study using Big Data latent dirichlet allocation (LDA) topic modelling," Journal of the Korea Institute of Information and Communication Engineering, vol. 25, no. 5, pp. 731-740, 2021. https://doi.org/10.6109/JKIICE.2021.25.5.731
  3. C. H. Lee, K. H. Kang, Y. H. Kim, H. N. Lim, J. H. Ku, and K. H. Kim, "A Study on the Factors of Well-aging through Big Data Analysis: Focusing on Newspaper Articles," Journal of the Korea Academia-Industrial cooperation Society, vol. 22, no. 5 pp. 354-360, 2021. https://doi.org/10.5762/KAIS.2021.22.5.354
  4. J. H. Lee, "Building an SNS Crawling System Using Python," Journal of the Korea Industrial Information Systems Research, vol. 23, no. 5, pp. 61-76, 2018. https://doi.org/10.9723/JKSIIS.2018.23.5.061
  5. C. Kohlschuer, P. Fankhauser, and W. Nejdl, "Boilerplate detection using shallow text features," in Proceedings of the third ACM international conference on Web Search and Data Mining (WSDM), New York: NY, pp. 441-450, 2010.
  6. W. M. Song and M. G. Kim, "Contents Extraction from HTML Documents using Text Block Context," Journal of KISS : Software and Applications, vol. 40, no. 3, pp. 155-163, 2013.
  7. H. G. Jeon and C. Koh, "Text Extraction Algorithm using the HTML Logical Structure Analysis," Journal of Digital Contents Society, vol. 16, no. 3, pp. 445-455, 2015. https://doi.org/10.9728/DCS.2015.16.3.445
  8. J. H. Mo and J. M. Yum "Korean Web Content Extraction using Tag Rank Position and Gradient Boosting," Journal of KIISE, vol. 44, no. 6, pp. 581-586, 2017. https://doi.org/10.5626/JOK.2017.44.6.581
  9. S. Wu, J. Liu, and J. Fan, "Automatic Web Content Extraction by Combination of Learning and Grouping," in Proceedings of the 24th International Conference on World Wide Web (WWW '15), pp. 1264-1274, 2015.
  10. S. H. Kim and H. J. Kim, "Logistic Regression Ensemble Method for Extracting Significant Information from Social Texts," KIPS Transactions on Software and Data Engineering, vol. 6, no. 5, pp. 279-284, 2017. https://doi.org/10.3745/KTSDE.2017.6.5.279
  11. T. Vogels, O. E. Ganea, and C. Eickhoff, "Web2text: Deep structured boilerplate removal," in Proceedings of the 40th European Conference on Information Retrieval, pp. 167-179, 2018.
  12. J. Leonhardt, A. Anand, and M. Khosla, "Boilerplate Removal using a Neural Sequence Labeling Model," in Companion Proceedings of the Web Conference 2020 (WWW '20), New York: NY, pp. 226-229, 2020.
  13. J. H. Kim and E. G. Kim, "HTML Text Extraction Using Frequency Analysis," Journal of the Korea Institute of Information and Communication Engineering, vol. 25, no. 9, 2021.
  14. Tharwat, A, "Classification assessment methods," Applied Computing and Informatics, vol. 17 no. 1, pp. 168-192, 2021. https://doi.org/10.1016/j.aci.2018.08.003