• Proceedings of the Korean Information Science Society Conference
• /
• 1999.10a
• /
• pp.93-95
• /
• 1999

최근 XML 문서에 대한 활용이 늘어나면서 이들 문서에 대한 저장 및 검색에 대한 요구가 증가하고 있다. XML문서는 SGML(Standard Generalized Markup Language) 문서가 가지고 있는 다양한 기능들과 구조적인 표현 능력, 그리고 사용의 용이성 등의 장점을 지닌 언어로 1996년 웹의 문서 표준으로 제안되었다. 따라서 XML 문서의 특성을 반영한 문서 검색시스템에 대한 요구가 시급한 상태이며, 기존의 시스템의 경우 구조 및 내용-기반 멀티미디어 문서검색을 효과적으로 지원하지 못하고 있다. 본 논문에서는 XML 문서의 구조정보 및 내용정보를 효과적으로 검색할 수 있는 XML 문서 저장 시스템을 설계 및 구현한다. 구현하는 시스템은 구조-기반 검색을 위해 o2store위에 역파일 인덱스를 구축하고 내용-기반 검색을 위해 X-tree를 사용한다. 또한 검색 인터페이스를 JAVA로 구현하여 효율적인 검색이 이루어지도록 한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2012.04a
• /
• pp.283-286
• /
• 2012

국가과학기술지식정보서비스(NTIS)는 국가 R&D와 관련된 과제, 성과, 인력, 시설 장비, 기술산업 정보에 대해 이용자에게 통합검색서비스를 제공한다. 이용자는 질의어를 입력하여 원하는 정보를 선별하게 되고, 한 건의 상세 메타정보 및 원문을 검색서비스의 최종 목적지로 삼는다. 이 때 이용 중인 정보와 유사한 다른 유형의 R&D 정보를 함께 제공한다면 이용자의 검색 및 탐색노력을 줄임으로써 정보획득의 요구를 쉽게 충족시킬 수 있다. 본 논문에서는 국가 R&D 정보의 메타데이터와 검색엔진의 부스팅 기법을 이용하여 이종 정보간 유사문서 검색 방법에 대해 논한다. 이는 이용자가 원하는 정보를 서비스 최종 화면(메타 상세보기)에서 제공함으로써 검색 서비스의 효율성을 증대시킨다.

• Annual Conference on Human and Language Technology
• /
• 2021.10a
• /
• pp.227-232
• /
• 2021

사실 검증(Fact verification) 문제는 문서 검색(Document retrieval), 증거 선택(Evidence selection), 증거 검증(Claim verification) 3가지 단계로 구성되어있다. 사실 검증 모델들의 주요 관심사인 증거 검증 단계에서 많은 모델이 제안되는 가운데 증거 선택 단계에 집중하여 강화 학습을 통해 해결한 모델이 제안되었다. 그래프 기반의 모델과 강화 학습 기반의 사실 검증 모델을 소개하고 각 모델을 한국어 사실 검증에 적용해본다. 또한, 두 모델을 같이 사용하여 각 모델의 장점을 가지는 부분을 병렬적으로 결합한 모델의 성능과 증거의 구성 단위에 따른 성능도 비교한다.

• Annual Conference on Human and Language Technology
• /
• 2023.10a
• /
• pp.399-403
• /
• 2023

본 논문에서는 다중 문서 기반의 대화 시스템을 통한 효율적인 정보 검색과 응답 생성에 중점을 둡니다. 대규모 데이터 집합에서 정확한 문서를 선택하는 데 필요한 검색의 중요성을 강조하며, 현재 검색 방법의 한계와 문제점을 지적합니다. 또한 더 자연스러운 답변을 생성하기 위해 대규모 언어 모델을 사용하게 되면서 fine-tuning 시에 발생하는 제약과 낭비를 모델의 제로샷 생성 능력을 활용하여 개선하려는 방안을 제안하며, 모델의 크기와 자원의 효율성에 대한 고려사항을 논의합니다. 우리의 접근 방식은 대규모 언어 모델을 프롬프트와 함께 다중 문서로 학습 없이 정보를 검색하고 응답을 생성하는 방향으로 접근하여 대화 시스템의 효율성과 유용성을 향상시킬 수 있음을 제시합니다.

• East Asian mathematical journal
• /
• v.40 no.3
• /
• pp.307-318
• /
• 2024

This paper explores the application of matrix factorization, specifically CUR decomposition, in the clustering of Korean language documents by topic. It addresses the unique challenges of Natural Language Processing (NLP) in dealing with the Korean language's distinctive features, such as agglutinative words and morphological ambiguity. The study compares the effectiveness of Latent Semantic Analysis (LSA) using CUR decomposition with the classical Singular Value Decomposition (SVD) method in the context of Korean text. Experiments are conducted using Korean Wikipedia documents and newspaper data, providing insight into the accuracy and efficiency of these techniques. The findings demonstrate the potential of CUR decomposition to improve the accuracy of document clustering in Korean, offering a valuable approach to text mining and information retrieval in agglutinative languages.

• The KIPS Transactions:PartD
• /
• v.12D no.5 s.101
• /
• pp.663-672
• /
• 2005

This Paper Proposes an approach that removes the redundancy of Path information and uses an inverted index, as an efficient way to store a large volume of XML documents and to retrieve wanted information from there. An XML document is decomposed into nodes based on its tree structure, and stored in relational tables according to the node type, with path information from the root to each node. The existing methods using path information store data for all element paths, which cause retrieval performance to be decreased with increased data volume. Our approach stores only data for leaf element path excluding internal element paths. As the inverted index is made by the leaf element path only, the number of posting lists by key words become smaller than those of the existing methods. For the storage and retrieval of U data, our approach doesn't require the XML schema information of XML documents and any extension of relational database. We demonstrate the better performance of on approach than the existing approaches within the scope of our experiment.

• Journal of Intelligence and Information Systems
• /
• v.17 no.3
• /
• pp.63-77
• /
• 2011

The World Wide Web is a very large distributed digital information space. From its origins in 1991, the web has grown to encompass diverse information resources as personal home pasges, online digital libraries and virtual museums. Some estimates suggest that the web currently includes over 500 billion pages in the deep web. The ability to search and retrieve information from the web efficiently and effectively is an enabling technology for realizing its full potential. With powerful workstations and parallel processing technology, efficiency is not a bottleneck. In fact, some existing search tools sift through gigabyte.syze precompiled web indexes in a fraction of a second. But retrieval effectiveness is a different matter. Current search tools retrieve too many documents, of which only a small fraction are relevant to the user query. Furthermore, the most relevant documents do not nessarily appear at the top of the query output order. Also, current search tools can not retrieve the documents related with retrieved document from gigantic amount of documents. The most important problem for lots of current searching systems is to increase the quality of search. It means to provide related documents or decrease the number of unrelated documents as low as possible in the results of search. For this problem, CiteSeer proposed the ACI (Autonomous Citation Indexing) of the articles on the World Wide Web. A "citation index" indexes the links between articles that researchers make when they cite other articles. Citation indexes are very useful for a number of purposes, including literature search and analysis of the academic literature. For details of this work, references contained in academic articles are used to give credit to previous work in the literature and provide a link between the "citing" and "cited" articles. A citation index indexes the citations that an article makes, linking the articleswith the cited works. Citation indexes were originally designed mainly for information retrieval. The citation links allow navigating the literature in unique ways. Papers can be located independent of language, and words in thetitle, keywords or document. A citation index allows navigation backward in time (the list of cited articles) and forwardin time (which subsequent articles cite the current article?) But CiteSeer can not indexes the links between articles that researchers doesn't make. Because it indexes the links between articles that only researchers make when they cite other articles. Also, CiteSeer is not easy to scalability. Because CiteSeer can not indexes the links between articles that researchers doesn't make. All these problems make us orient for designing more effective search system. This paper shows a method that extracts subject and predicate per each sentence in documents. A document will be changed into the tabular form that extracted predicate checked value of possible subject and object. We make a hierarchical graph of a document using the table and then integrate graphs of documents. The graph of entire documents calculates the area of document as compared with integrated documents. We mark relation among the documents as compared with the area of documents. Also it proposes a method for structural integration of documents that retrieves documents from the graph. It makes that the user can find information easier. We compared the performance of the proposed approaches with lucene search engine using the formulas for ranking. As a result, the F.measure is about 60% and it is better as about 15%.

• Journal of KIISE:Software and Applications
• /
• v.29 no.4
• /
• pp.270-277
• /
• 2002

Text filtering is a task of deciding whether a document has relevance to a specified topic. As Internet and Web becomes wide-spread and the number of documents delivered by e-mail explosively grows the importance of text filtering increases as well. The aim of this paper is to improve the accuracy of text filtering systems by using machine learning techniques. We apply AdaBoost algorithms to the filtering task. An AdaBoost algorithm generates and combines a series of simple hypotheses. Each of the hypotheses decides the relevance of a document to a topic on the basis of whether or not the document includes a certain word. We begin with an existing AdaBoost algorithm which uses weak hypotheses with their output of 1 or -1. Then we extend the algorithm to use weak hypotheses with real-valued outputs which was proposed recently to improve error reduction rates and final filtering performance. Next, we attempt to achieve further improvement in the AdaBoost's performance by first setting weights randomly according to the continuous Poisson distribution, executing AdaBoost, repeating these steps several times, and then combining all the hypotheses learned. This has the effect of mitigating the ovefitting problem which may occur when learning from a small number of data. Experiments have been performed on the real document collections used in TREC-8, a well-established text retrieval contest. This dataset includes Financial Times articles from 1992 to 1994. The experimental results show that AdaBoost with real-valued hypotheses outperforms AdaBoost with binary-valued hypotheses, and that AdaBoost iterated with random weights further improves filtering accuracy. Comparison results of all the participants of the TREC-8 filtering task are also provided.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.14 no.4
• /
• pp.1400-1418
• /
• 2020

In the development of commercial promotion, chatbot is known as one of significant skill by application of natural language processing (NLP). Conventional design methods are using bag-of-words model (BOW) alone based on Google database and other online corpus. For one thing, in the bag-of-words model, the vectors are Irrelevant to one another. Even though this method is friendly to discrete features, it is not conducive to the machine to understand continuous statements due to the loss of the connection between words in the encoded word vector. For other thing, existing methods are used to test in state-of-the-art online corpus but it is hard to apply in real applications such as telemarketing data. In this paper, we propose an improved chatbot design way using hybrid bag-of-words model and skip-gram model based on the real telemarketing data. Specifically, we first collect the real data in the telemarketing field and perform data cleaning and data classification on the constructed corpus. Second, the word representation is adopted hybrid bag-of-words model and skip-gram model. The skip-gram model maps synonyms in the vicinity of vector space. The correlation between words is expressed, so the amount of information contained in the word vector is increased, making up for the shortcomings caused by using bag-of-words model alone. Third, we use the term frequency-inverse document frequency (TF-IDF) weighting method to improve the weight of key words, then output the final word expression. At last, the answer is produced using hybrid retrieval model and generate model. The retrieval model can accurately answer questions in the field. The generate model can supplement the question of answering the open domain, in which the answer to the final reply is completed by long-short term memory (LSTM) training and prediction. Experimental results show which the hybrid word vector expression model can improve the accuracy of the response and the whole system can communicate with humans.

• The Journal of the Korea Contents Association
• /
• v.6 no.10
• /
• pp.134-142
• /
• 2006

In this paper, we investigate a Hausdorff distance, which is used for the measurement of image similarity, to see whether it is also effective for document retrieval. The proposed method uses a local crowdedness and a Hausdorff distance to locate text images by determining whether a pair of images scanned at different time comes from the same text or not. To reduce the processing time, which is one of the disadvantages of a Hausdorff distance algorithm, we adopt a local crowdedness for feature point extraction. We apply the proposed method to 190 pairs of the same class and 190 pairs of the different class collected from postal envelop images. The results show that the modified Hausdorff distance proposed in this paper performed well in locating the tort region and calculating the degree of similarity between two images. An improvement of accuracy by 2.7% and 9.0% has been obtained, compared to a binary correlation method and the original Hausdorff distance method, respectively.