Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Information Extraction System from Multi Source Unstructured Documents for Knowledge Base Expansion

지식베이스 확장을 위한 멀티소스 비정형 문서에서의 정보 추출 시스템의 개발

  • Choi, Hyunseung (Department of Industrial Engineering, Yonsei University) ;
  • Kim, Mintae (Department of Industrial Engineering, Yonsei University) ;
  • Kim, Wooju (Department of Industrial Engineering, Yonsei University) ;
  • Shin, Dongwook (Knowledge Technology Cell, AI Technology Unit, AI Center, SK Telecom) ;
  • Lee, Yong Hun (Knowledge Technology Cell, AI Technology Unit, AI Center, SK Telecom)
  • Received : 2018.10.29
  • Accepted : 2018.12.18
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a methodology to extract answer information about queries from various types of unstructured documents collected from multi-sources existing on web in order to expand knowledge base. The proposed methodology is divided into the following steps. 1) Collect relevant documents from Wikipedia, Naver encyclopedia, and Naver news sources for "subject-predicate" separated queries and classify the proper documents. 2) Determine whether the sentence is suitable for extracting information and derive the confidence. 3) Based on the predicate feature, extract the information in the proper sentence and derive the overall confidence of the information extraction result. In order to evaluate the performance of the information extraction system, we selected 400 queries from the artificial intelligence speaker of SK-Telecom. Compared with the baseline model, it is confirmed that it shows higher performance index than the existing model. The contribution of this study is that we develop a sequence tagging model based on bi-directional LSTM-CRF using the predicate feature of the query, with this we developed a robust model that can maintain high recall performance even in various types of unstructured documents collected from multiple sources. The problem of information extraction for knowledge base extension should take into account heterogeneous characteristics of source-specific document types. The proposed methodology proved to extract information effectively from various types of unstructured documents compared to the baseline model. There is a limitation in previous research that the performance is poor when extracting information about the document type that is different from the training data. In addition, this study can prevent unnecessary information extraction attempts from the documents that do not include the answer information through the process for predicting the suitability of information extraction of documents and sentences before the information extraction step. It is meaningful that we provided a method that precision performance can be maintained even in actual web environment. The information extraction problem for the knowledge base expansion has the characteristic that it can not guarantee whether the document includes the correct answer because it is aimed at the unstructured document existing in the real web. When the question answering is performed on a real web, previous machine reading comprehension studies has a limitation that it shows a low level of precision because it frequently attempts to extract an answer even in a document in which there is no correct answer. The policy that predicts the suitability of document and sentence information extraction is meaningful in that it contributes to maintaining the performance of information extraction even in real web environment. The limitations of this study and future research directions are as follows. First, it is a problem related to data preprocessing. In this study, the unit of knowledge extraction is classified through the morphological analysis based on the open source Konlpy python package, and the information extraction result can be improperly performed because morphological analysis is not performed properly. To enhance the performance of information extraction results, it is necessary to develop an advanced morpheme analyzer. Second, it is a problem of entity ambiguity. The information extraction system of this study can not distinguish the same name that has different intention. If several people with the same name appear in the news, the system may not extract information about the intended query. In future research, it is necessary to take measures to identify the person with the same name. Third, it is a problem of evaluation query data. In this study, we selected 400 of user queries collected from SK Telecom 's interactive artificial intelligent speaker to evaluate the performance of the information extraction system. n this study, we developed evaluation data set using 800 documents (400 questions * 7 articles per question (1 Wikipedia, 3 Naver encyclopedia, 3 Naver news) by judging whether a correct answer is included or not. To ensure the external validity of the study, it is desirable to use more queries to determine the performance of the system. This is a costly activity that must be done manually. Future research needs to evaluate the system for more queries. It is also necessary to develop a Korean benchmark data set of information extraction system for queries from multi-source web documents to build an environment that can evaluate the results more objectively.

지식베이스를 구축하는 작업은 도메인 전문가가 온톨로지 스키마를 이해한 뒤, 직접 지식을 정제하는 수작업이 요구되는 만큼 비용이 많이 드는 활동이다. 이에, 도메인 전문가 없이 다양한 웹 환경으로부터 질의에 대한 답변 정보를 추출하기 위한 자동화된 시스템의 연구개발의 필요성이 제기되고 있다. 기존의 정보 추출 관련 연구들은 웹에 존재하는 다양한 형태의 문서 중 학습데이터와 상이한 형태의 문서에서는 정보를 효과적으로 추출하기 어렵다는 한계점이 존재한다. 또한, 기계 독해와 관련된 연구들은 문서에 정답이 있는 경우를 가정하고 질의에 대한 답변정보를 추출하는 경우로서, 문서의 정답포함 여부를 보장할 수 없는 실제 웹의 비정형 문서로부터의 정보추출에서는 낮은 성능을 보인다는 한계점이 존재한다. 본 연구에서는 지식베이스 확장을 위하여 웹에 존재하는 멀티소스 비정형 문서로부터 질의에 대한 정보를 추출하기 위한 시스템의 개발 방법론을 제안하고자 한다. 본 연구에서 제안한 방법론은 "주어(Subject)-서술어(Predicate)"로 구분된 질의에 대하여 위키피디아, 네이버 백과사전, 네이버 뉴스 3개 웹 소스로부터 수집된 비정형 문서로부터 관련 정보를 추출하며, 제안된 방법론을 적용한 시스템의 성능평가를 위하여, Wu and Weld(2007)의 모델을 베이스라인 모델로 선정하여 성능을 비교분석 하였다. 연구결과 제안된 모델이 베이스라인 모델에 비해, 위키피디아, 네이버 백과사전, 네이버 뉴스 등 다양한 형태의 문서에서 정보를 효과적으로 추출하는 강건한 모델임을 입증하였다. 본 연구의 결과는 현업 지식베이스 관리자에게 지식베이스 확장을 위한 웹에서 질의에 대한 답변정보를 추출하기 위한 시스템 개발의 지침서로서 실무적인 시사점을 제공함과 동시에, 추후 다양한 형태의 질의응답 시스템 및 정보추출 연구로의 확장에 기여할 수 있을 것으로 기대한다.

Keywords

JJSHBB_2018_v24n4_111_f0001.png 이미지

Proposed Information Extraction Methodology

JJSHBB_2018_v24n4_111_f0002.png 이미지

Architecture of Confidence Calculation Model Based on Convolutional Neural Network

JJSHBB_2018_v24n4_111_f0003.png 이미지

Architecture of the Query’s Predicate Based Information Extraction Model

Document Collection & Proper Document Classification

JJSHBB_2018_v24n4_111_t0001.png 이미지

Proper Sentence Classification

JJSHBB_2018_v24n4_111_t0002.png 이미지

Data set of Proper Sentence Confidence Measure Model

JJSHBB_2018_v24n4_111_t0003.png 이미지

Training Parameters Proper Sentence Confidence Measure Model

JJSHBB_2018_v24n4_111_t0004.png 이미지

Train Performance of Proper Sentence Confidence Measure Model

JJSHBB_2018_v24n4_111_t0005.png 이미지

Validation Performance of Proper Sentence Confidence Measure Model

JJSHBB_2018_v24n4_111_t0006.png 이미지

Test Performance of Proper Sentence Confidence Measure Model

JJSHBB_2018_v24n4_111_t0007.png 이미지

Feature Description of Bi-directional LSTM CRF Model

JJSHBB_2018_v24n4_111_t0008.png 이미지

Post Processing Dictionary

JJSHBB_2018_v24n4_111_t0009.png 이미지

Data Set Description for Information Extraction Model

JJSHBB_2018_v24n4_111_t0010.png 이미지

Training Parameters Proper Information Extraction Model

JJSHBB_2018_v24n4_111_t0011.png 이미지

Train Performance of Information Extraction Model

JJSHBB_2018_v24n4_111_t0012.png 이미지

Train Performance of Information Extraction Model

JJSHBB_2018_v24n4_111_t0013.png 이미지

Test Performance of Information Extraction Model

JJSHBB_2018_v24n4_111_t0014.png 이미지

Query Set Description

JJSHBB_2018_v24n4_111_t0015.png 이미지

Confusion Matrix for Proposed Information Extraction

JJSHBB_2018_v24n4_111_t0016.png 이미지

Evaluation Metric for Proposed Information Extraction

JJSHBB_2018_v24n4_111_t0017.png 이미지

Performance Measure on Wikipedia

JJSHBB_2018_v24n4_111_t0018.png 이미지

Performance Measure on Naver Encyclopedia

JJSHBB_2018_v24n4_111_t0019.png 이미지

Performance Measure on Naver News

JJSHBB_2018_v24n4_111_t0020.png 이미지

Performance Measure on Multi Source

JJSHBB_2018_v24n4_111_t0021.png 이미지

References

  1. Ahn, S., "Deep Learning Architectures and Applications", Journal of Intelligence and Information Systems, Vol.22, No.2(2018), 127-142. https://doi.org/10.13088/JIIS.2016.22.2.127
  2. Banko, M., M. J. Cafarella, S. Soderland, M. Broadhead, O. Etzioni, "Open information extraction from the web." IJCAI. Vol. 7, (2007), 2670-2676.
  3. Eikvil, L, "Information extraction from world wide web-a survey.", Technical Report 945, Norwegian Computing Center, 1999.
  4. Etzioni, O., M. Cafarella, and D. Downey, "Web-scale information extraction in knowitall:(preliminary results)." Proceedings of the 13th international conference on World Wide Web. ACM, (2004), 100-110.
  5. Gaizauskas, R., and Y. Wilks, "Information extraction: Beyond document retrieval.", Journal of documentation, Vol 54, No.1 (1998), 70-105. https://doi.org/10.1108/EUM0000000007162
  6. Hermann, K., et al. "Teaching machines to read and comprehend." Advances in Neural Information Processing Systems, (2015), 1693-1701.
  7. Huang, Z., W. Xu, and K. Yu, "Bidirectional LSTM-CRF models for sequence tagging." arXiv preprint, (2015).
  8. JIN, S., H. Jang, W. KIM, "Improving Bidirectional LSTM-CRF model Of Sequence Tagging by using Ontology knowledge based feature", Journal of Intelligence and Information Systems, Vol.24, No.1(2018), 253-267). https://doi.org/10.13088/JIIS.2018.24.1.253
  9. Khot, T., A. Sabharwal, and P. Clark, "Answering complex questions using open information extraction." arXiv preprint, (2017).
  10. Kim, Y. "Convolutional neural networks for sentence classification." arXiv preprint, (2014).
  11. Park, E.L., and Cho. S, "KoNLPy: Korean natural language processing in Python." Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology, (2014). 133-136.
  12. Park, H., M. Song, K. Shin, " Sentiment Analysis of Korean Reviews Using CNN: Focusing on Morpheme Embedding", Journal of Intelligence and Information Systems, Vol.24, No.2(2018), 59-83. https://doi.org/10.13088/JIIS.2018.24.2.059
  13. Qiu , Lin., H. Zhou, Y. Q, W. Zhang, S. Li, S. Rong, D. Ru, L. Qian, W. Tu, Y. Yu, "QA4IE: A Question Answering based Framework for Information Extraction." arXiv preprint, (2018).
  14. Rajpurkar, P., J. Zhang, K. Lopyrev, P. Liang, "Squad: 100,000+ questions for machine comprehension of text." arXiv preprint, (2016).
  15. Seo, M., A. Kembhavi, A. Farhadi, H. Hajishirzi, "Bidirectional attention flow for machine comprehension." arXiv preprint, (2016).
  16. Wang, W., N. Yang, F. Wei, B. Chang, M. Zhou, "Gated Self-matching networks for reading comprehension and question answering." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, (2017). 189-198.
  17. Wang, S., and J. Jiang, "Machine comprehension using match-lstm and answer pointer." arXiv preprint, (2016).
  18. Wu, F., and D. S. Weld, "Autonomously semantifying wikipedia." Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, ACM, (2007), 41-50.