• Annual Conference on Human and Language Technology
• /
• 2001.10d
• /
• pp.157-162
• /
• 2001

한국어의 경우 띄어쓰기의 자유로움과 명사들이 비교적 자유롭게 결합하여 새로운 복합명사(compound noun)를 형성한다. 따라서, 정보검색에서 복합명사를 적절하게 처리하게 되면 검색 효율을 향상시킬 수 있다. 본 논문에서는 질의에 포함된 단일명사, 복합명사, 그리고 복합명사를 이루는 구성명사의 적절한 가중치 부여 방법에 대하여 기술한다. 일반적인 tf*idf가중치 방법은 문서 내 빈도수(tf)만을 강조하여 문서 내 발생빈도가 낮은 복합명사의 경우 낮은 가중치를 갖는다. 반대로, 역문헌 빈도수(idf)로 인해 복합명사가 단일명사보다 높은 가중치를 갖게 되면 단일명사의 가중치를 지나치게 떨어뜨려 검색 성능을 저하시킨다. 이런 문제를 해결하기 위해서 복합명사의 통계적인 특성을 고려하고, 복합명사를 이루는 구성명사의 적절한 가중치 사용과 tf*idf 변화 범위에 따른 파라메터를 이용하였다. 결과적으로 본 논문에서는 질의 색인어의 종류에 따라 가중치를 달리 부여함으로써 검색 성능을 향상시킬 수 있는 가중치 부여 방법을 제시하고 검증 실험을 통해 유효성을 제시했다는 점에서 그 의의가 있다고 하겠다.

• Proceedings of the Korea Inteligent Information System Society Conference
• /
• 2004.11a
• /
• pp.349-357
• /
• 2004

최근 메일이 커뮤니케이션의 중요한 수단 중 하나로 자리잡고 있으나 과다한 정보 전달 및 원하는 않는 정보의 전달 등으로 인해 사용자가 메일을 확인하고 정리하기 위해 많은 시간과 노력을 투자하고 있다. 본 연구에서는 사용자가 적은 시간과 노력으로 메일을 활용하고, 보다 편리하게 사용할 수 있는 폴더 추천 방법론 개발을 목표로 하고 있다. 이러한 목표를 위해 TF-IDF를 기반으로 하는 다양한 방법론이 개발되고 활용되어 왔으나, 메일이라는 영역의 특성상 단어의 수나 내용에 한계가 있는 경우 안정적인 추천이 이루어지지 못할 수 있었다. 따라서 본 연구에서는 기존의 TF-IDF 방법에 사용자의 지식을 부여한 새로운 방법을 제시함으로써 단어의 수나 내용에 한계가 있는 경우에도 안정적인 추천이 이루어질 수 있도록 하였다. 또한 실제 데이터를 활용하여 기존의 방법과 본 연구에서 제시한 방법론을 비교 실험해 봄으로써, 본 연구에서 제시하고 있는 방법론의 성능을 입증하고자 하였다.

• Proceedings of the Korean Information Science Society Conference
• /
• 2010.06c
• /
• pp.349-352
• /
• 2010

질의를 이용한 정보 검색 기술에서 단어 의미의 모호성에 의해 사용자가 검색 하고자 하는 주제 이외의 문서 까지 검색되고 있다. 이러한 문제는 모바일기기의 검색 환경에서 두드러진다. 모바일에서의 검색은 문서의 로딩속도가 느리며 작은 화면에 의해 스크롤이 잦다. 그러므로 원하는 검색 결과가 검색 첫 페이지 이외에 위치하거나, 또는 페이지 하단에 위치할 경우 검색 결과를 확인하는 대에 많은 시간과 노력이 필요하다. 이러한 문제를 해결하기위해선 단어 의미의 모호성을 해결하고 사용자가 검색하고자하는 주제의 검색결과를 검색 상위에 위치시킬 수 있는 방법을 필요로 한다. 이 연구에서는 연관 단어 추출과 TF*IDF를 이용하여, 검색결과를 re-ranking하는 방법을 제시한다.

• Journal of Software Assessment and Valuation
• /
• v.12 no.2
• /
• pp.1-8
• /
• 2016

In this paper, we study abstracting and identifying license file from a package to prevent unintentional intellectual property infringement because of lost/modified/confliction of license information when redistributing open source software. To invest character of the license files, we analyzed 322 licenses by n-gram and TF-IDF methods, and abstract license files from the packages. We identified license information with a similarity of the registered licenses by cosine measurement.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.05a
• /
• pp.353-354
• /
• 2023

디지털 환경이 진화함에 따라 가짜뉴스가 늘어나고 있다. 이를 판별하기 위해 법적 규제에 대한 논의가 있으나, 가짜뉴스에 대한 범위와 정의가 명확하지 않아 규제가 쉽지 않다. 본 논문에서는 이에 대한 대안으로 TF-IDF 기법과 KoBERT 모델을 이용한 키워드 추출 및 문장 유사도 분석을 통해 YouTube 플랫폼을 대상으로 한 가짜뉴스 판별을 위한 모델을 제안한다.

• Journal of the Korean Data Analysis Society
• /
• v.20 no.6
• /
• pp.2917-2932
• /
• 2018

Many researches are accomplished as a result of the efforts of developing the production predicting model to solve the supply imbalance of onions which are vegetables very closely related to Korean food. But considering the possibility of storing onions, it is very difficult to solve the supply imbalance of onions only with predicting the production. So, this paper's purpose is trying to build a sentiment dictionary to predict the price of onions by using the internet articles which include the informations about the production of onions and various factors of the price, and these articles are very easy to access on our daily lives. Articles about onions are from 2012 to 2016, using TF-IDF for comparing with four kinds of TF-IDFs through the documents classification of wholesale prices of onions. As a result of classifying the positive/negative words for price by k-means clustering, DBSCAN (density based spatial cluster application with noise) clustering, GMM (Gaussian mixture model) clustering which are partitional clustering, GMM clustering is composed with three meaningful dictionaries. To compare the reasonability of these built dictionary, applying classified articles about the rise and drop of the price on logistic regression, and it shows 85.7% accuracy.

• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.3
• /
• pp.145-154
• /
• 2016

Because of the recent interest in the u-Health and development of IT technology, a need of utilizing a medical information data has been increased. Among previous studies that utilize various data mining algorithms for processing medical information data, there are studies of association rule analysis. In the studies, an association between the symptoms with specified diseases is the target to discover, however, infrequent terms which can be important information for a disease diagnosis are not considered in most cases. In this paper, we proposed a new association rule mining system considering the importance of each term using TF-IDF weight to consider infrequent but important items. In addition, the proposed system can predict candidate diagnoses from medical text records using term similarity analysis based on medical ontology.

• Journal of the Korea Society of Computer and Information
• /
• v.20 no.2
• /
• pp.121-129
• /
• 2015

With the explosive growth of information about books, there is a growing number of customers who find it difficult to pick a book. Against the backdrop, the importance of a book recommendation system becomes greater, through which appropriate information about books could be offered then to encourage customers to buy a book in the end. However, existing recommendation systems based on the bibliographical information or user data reveal the reliability issue found in their recommendation results. This is why it is necessary to reflect semantic information extracted from the texts of a book's main body in a recommendation system. Accordingly, this paper suggests a method for extracting keywords from the main body of novels, as a preceding research, by using TF-IDF method as well as the text structure. To this end, the texts of 100 novels have been collected then to divide them into four structural elements of preface, dialogue, non-dialogue and closing. Then, the TF-IDF weight of each keyword has been calculated. The calculation results show that the extraction accuracy of keywords improves by 42.1% in performance when more weight is given to dialogue while including preface and closing instead of using just the main body.

• Journal of Korea Multimedia Society
• /
• v.22 no.3
• /
• pp.374-385
• /
• 2019

A Web Crawler is a program, which is commonly used by search engines to find the new brainchild on the internet. The use of crawlers has made the web easier for users. In this paper, we have used unstructured data by structuralization to collect data from the web pages. Our system is able to choose the word near our keyword in more than one document using unstructured way. Neighbor data were collected on the keyword through word2vec. The system goal is filtered at the data acquisition level and for a large taxonomy. The main problem in text taxonomy is how to improve the classification accuracy. In order to improve the accuracy, we propose a new weighting method of TF-IDF. In this paper, we modified TF-algorithm to calculate the accuracy of unstructured data. Finally, our system proposes a competent web pages search crawling algorithm, which is derived from TF-IDF and RL Web search algorithm to enhance the searching efficiency of the relevant information. In this paper, an attempt has been made to research and examine the work nature of crawlers and crawling algorithms in search engines for efficient information retrieval.

• The Korean Journal of Applied Statistics
• /
• v.33 no.5
• /
• pp.615-625
• /
• 2020

The document-term frequency matrix is a term extracted from documents in which the group information exists in text mining. In this study, we generated the document-term frequency matrix for document classification according to research field. We applied the traditional term weighting function term frequency-inverse document frequency (TF-IDF) to the generated document-term frequency matrix. In addition, we applied term frequency-inverse gravity moment (TF-IGM). We also generated a document-keyword weighted matrix by extracting keywords to improve the document classification accuracy. Based on the keywords matrix extracted, we classify documents using a deep neural network. In order to find the optimal model in the deep neural network, the accuracy of document classification was verified by changing the number of hidden layers and hidden nodes. Consequently, the model with eight hidden layers showed the highest accuracy and all TF-IGM document classification accuracy (according to parameter changes) were higher than TF-IDF. In addition, the deep neural network was confirmed to have better accuracy than the support vector machine. Therefore, we propose a method to apply TF-IGM and a deep neural network in the document classification.