• Title/Summary/Keyword: Automatic Summarization

Search Result 95, Processing Time 0.027 seconds

Semantic Pre-training Methodology for Improving Text Summarization Quality (텍스트 요약 품질 향상을 위한 의미적 사전학습 방법론)

  • Mingyu Jeon;Namgyu Kim
    • Smart Media Journal
    • /
    • v.12 no.5
    • /
    • pp.17-27
    • /
    • 2023
  • Recently, automatic text summarization, which automatically summarizes only meaningful information for users, is being studied steadily. Especially, research on text summarization using Transformer, an artificial neural network model, has been mainly conducted. Among various studies, the GSG method, which trains a model through sentence-by-sentence masking, has received the most attention. However, the traditional GSG has limitations in selecting a sentence to be masked based on the degree of overlap of tokens, not the meaning of a sentence. Therefore, in this study, in order to improve the quality of text summarization, we propose SbGSG (Semantic-based GSG) methodology that selects sentences to be masked by GSG considering the meaning of sentences. As a result of conducting an experiment using 370,000 news articles and 21,600 summaries and reports, it was confirmed that the proposed methodology, SbGSG, showed superior performance compared to the traditional GSG in terms of ROUGE and BERT Score.

Automatic Genre Classification of Sports News Video Using Features of Playfield and Motion Vector (필드와 모션벡터의 특징정보를 이용한 스포츠 뉴스 비디오의 장르 분류)

  • Song, Mi-Young;Jang, Sang-Hyun;Cho, Hyung-Je
    • The KIPS Transactions:PartB
    • /
    • v.14B no.2
    • /
    • pp.89-98
    • /
    • 2007
  • For browsing, searching, and manipulating video documents, an indexing technique to describe video contents is required. Until now, the indexing process is mostly carried out by specialists who manually assign a few keywords to the video contents and thereby this work becomes an expensive and time consuming task. Therefore, automatic classification of video content is necessary. We propose a fully automatic and computationally efficient method for analysis and summarization of spots news video for 5 spots news video such as soccer, golf, baseball, basketball and volleyball. First of all, spots news videos are classified as anchor-person Shots, and the other shots are classified as news reports shots. Shot classification is based on image preprocessing and color features of the anchor-person shots. We then use the dominant color of the field and motion features for analysis of sports shots, Finally, sports shots are classified into five genre type. We achieved an overall average classification accuracy of 75% on sports news videos with 241 scenes. Therefore, the proposed method can be further used to search news video for individual sports news and sports highlights.

Automatic Text Summarization based on Selective Copy mechanism against for Addressing OOV (미등록 어휘에 대한 선택적 복사를 적용한 문서 자동요약)

  • Lee, Tae-Seok;Seon, Choong-Nyoung;Jung, Youngim;Kang, Seung-Shik
    • Smart Media Journal
    • /
    • v.8 no.2
    • /
    • pp.58-65
    • /
    • 2019
  • Automatic text summarization is a process of shortening a text document by either extraction or abstraction. The abstraction approach inspired by deep learning methods scaling to a large amount of document is applied in recent work. Abstractive text summarization involves utilizing pre-generated word embedding information. Low-frequent but salient words such as terminologies are seldom included to dictionaries, that are so called, out-of-vocabulary(OOV) problems. OOV deteriorates the performance of Encoder-Decoder model in neural network. In order to address OOV words in abstractive text summarization, we propose a copy mechanism to facilitate copying new words in the target document and generating summary sentences. Different from the previous studies, the proposed approach combines accurate pointing information and selective copy mechanism based on bidirectional RNN and bidirectional LSTM. In addition, neural network gate model to estimate the generation probability and the loss function to optimize the entire abstraction model has been applied. The dataset has been constructed from the collection of abstractions and titles of journal articles. Experimental results demonstrate that both ROUGE-1 (based on word recall) and ROUGE-L (employed longest common subsequence) of the proposed Encoding-Decoding model have been improved to 47.01 and 29.55, respectively.

A Study on the Effect of the Document Summarization Technique on the Fake News Detection Model (문서 요약 기법이 가짜 뉴스 탐지 모형에 미치는 영향에 관한 연구)

  • Shim, Jae-Seung;Won, Ha-Ram;Ahn, Hyunchul
    • Journal of Intelligence and Information Systems
    • /
    • v.25 no.3
    • /
    • pp.201-220
    • /
    • 2019
  • Fake news has emerged as a significant issue over the last few years, igniting discussions and research on how to solve this problem. In particular, studies on automated fact-checking and fake news detection using artificial intelligence and text analysis techniques have drawn attention. Fake news detection research entails a form of document classification; thus, document classification techniques have been widely used in this type of research. However, document summarization techniques have been inconspicuous in this field. At the same time, automatic news summarization services have become popular, and a recent study found that the use of news summarized through abstractive summarization has strengthened the predictive performance of fake news detection models. Therefore, the need to study the integration of document summarization technology in the domestic news data environment has become evident. In order to examine the effect of extractive summarization on the fake news detection model, we first summarized news articles through extractive summarization. Second, we created a summarized news-based detection model. Finally, we compared our model with the full-text-based detection model. The study found that BPN(Back Propagation Neural Network) and SVM(Support Vector Machine) did not exhibit a large difference in performance; however, for DT(Decision Tree), the full-text-based model demonstrated a somewhat better performance. In the case of LR(Logistic Regression), our model exhibited the superior performance. Nonetheless, the results did not show a statistically significant difference between our model and the full-text-based model. Therefore, when the summary is applied, at least the core information of the fake news is preserved, and the LR-based model can confirm the possibility of performance improvement. This study features an experimental application of extractive summarization in fake news detection research by employing various machine-learning algorithms. The study's limitations are, essentially, the relatively small amount of data and the lack of comparison between various summarization technologies. Therefore, an in-depth analysis that applies various analytical techniques to a larger data volume would be helpful in the future.

Subject-Balanced Intelligent Text Summarization Scheme (주제 균형 지능형 텍스트 요약 기법)

  • Yun, Yeoil;Ko, Eunjung;Kim, Namgyu
    • Journal of Intelligence and Information Systems
    • /
    • v.25 no.2
    • /
    • pp.141-166
    • /
    • 2019
  • Recently, channels like social media and SNS create enormous amount of data. In all kinds of data, portions of unstructured data which represented as text data has increased geometrically. But there are some difficulties to check all text data, so it is important to access those data rapidly and grasp key points of text. Due to needs of efficient understanding, many studies about text summarization for handling and using tremendous amounts of text data have been proposed. Especially, a lot of summarization methods using machine learning and artificial intelligence algorithms have been proposed lately to generate summary objectively and effectively which called "automatic summarization". However almost text summarization methods proposed up to date construct summary focused on frequency of contents in original documents. Those summaries have a limitation for contain small-weight subjects that mentioned less in original text. If summaries include contents with only major subject, bias occurs and it causes loss of information so that it is hard to ascertain every subject documents have. To avoid those bias, it is possible to summarize in point of balance between topics document have so all subject in document can be ascertained, but still unbalance of distribution between those subjects remains. To retain balance of subjects in summary, it is necessary to consider proportion of every subject documents originally have and also allocate the portion of subjects equally so that even sentences of minor subjects can be included in summary sufficiently. In this study, we propose "subject-balanced" text summarization method that procure balance between all subjects and minimize omission of low-frequency subjects. For subject-balanced summary, we use two concept of summary evaluation metrics "completeness" and "succinctness". Completeness is the feature that summary should include contents of original documents fully and succinctness means summary has minimum duplication with contents in itself. Proposed method has 3-phases for summarization. First phase is constructing subject term dictionaries. Topic modeling is used for calculating topic-term weight which indicates degrees that each terms are related to each topic. From derived weight, it is possible to figure out highly related terms for every topic and subjects of documents can be found from various topic composed similar meaning terms. And then, few terms are selected which represent subject well. In this method, it is called "seed terms". However, those terms are too small to explain each subject enough, so sufficient similar terms with seed terms are needed for well-constructed subject dictionary. Word2Vec is used for word expansion, finds similar terms with seed terms. Word vectors are created after Word2Vec modeling, and from those vectors, similarity between all terms can be derived by using cosine-similarity. Higher cosine similarity between two terms calculated, higher relationship between two terms defined. So terms that have high similarity values with seed terms for each subjects are selected and filtering those expanded terms subject dictionary is finally constructed. Next phase is allocating subjects to every sentences which original documents have. To grasp contents of all sentences first, frequency analysis is conducted with specific terms that subject dictionaries compose. TF-IDF weight of each subjects are calculated after frequency analysis, and it is possible to figure out how much sentences are explaining about each subjects. However, TF-IDF weight has limitation that the weight can be increased infinitely, so by normalizing TF-IDF weights for every subject sentences have, all values are changed to 0 to 1 values. Then allocating subject for every sentences with maximum TF-IDF weight between all subjects, sentence group are constructed for each subjects finally. Last phase is summary generation parts. Sen2Vec is used to figure out similarity between subject-sentences, and similarity matrix can be formed. By repetitive sentences selecting, it is possible to generate summary that include contents of original documents fully and minimize duplication in summary itself. For evaluation of proposed method, 50,000 reviews of TripAdvisor are used for constructing subject dictionaries and 23,087 reviews are used for generating summary. Also comparison between proposed method summary and frequency-based summary is performed and as a result, it is verified that summary from proposed method can retain balance of all subject more which documents originally have.

Analysis on Automatic Summarization Functions of the Single Document and the Multi Documents (단일문서와 복수문서 자동요약의 특성에 따른 기능 분석)

  • 최상희
    • Proceedings of the Korean Society for Information Management Conference
    • /
    • 2003.08a
    • /
    • pp.303-312
    • /
    • 2003
  • 요약은 원문의 주제를 파악하여 원문의 축약판을 만들어 이용자에게 제공하는 중요한 정보 생산 과정이다. 최근 이용자에게 제공되는 정보량이 급증하면서 자동 요약에 대한 필요성이 더욱 증가하고 있으며 단일문서의 내용을 파악하는 도구로써 활용되던 요약이 문서집합의 내용을 파악하는 도구 및 새로운 정보생성 수단으로 그 기능을 넓혀가고 있다. 본 논고에서는 자동요약의 기본 개념과 요약대상의 문서 수에 따른 요약 특성 및 기능을 고찰하였다.

  • PDF

Automatic Text Summarization Using Thesaurus (시소러스를 이용한 문서 자동 요약)

  • 이창범;박혁로
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2001.04b
    • /
    • pp.352-354
    • /
    • 2001
  • 문서 자동요약은 입력된 문서에 대해 컴퓨터가 자동으로 요약을 생성하는 과정을 의미한다. 즉, 컴퓨터가 문서의 기본적인 내용을 유지하면서 문서의 복잡도 즉 문서의 길이를 줄이는 작업이다. 효율적인 정보 접근을 제공함과 동시에 정보 과적재를 해결하기 하기 위한 하나의 방법으로 문서 자동요약에 관한 연구가 활발히 진행되고 있다. 본 논문에서는 의미기반 정보검색용 시소러스(thesaurus)를 이용한 문서 자동요약을 제안한다. 제안한 방법에서는 단어간의 연관 관계 즉, 동의어, 유의어, 상위어, 하위어 관계를 문서 요약에 이용한다. 크게 연관 사슬 형성 단계, 중심 문장 추출 단계, 요약 생성 단계의 새단계로 나누어 요약을 생성한다. 수동 요약된 신문기사를 대상으로 평가한 결과 평균 66%가 일치하였다.

  • PDF

World Co-occurrence based Automatic Text Summarization (단어공기정보를 이용한 자동화 문서 요약)

  • 류동원;이종혁
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2000.04b
    • /
    • pp.345-347
    • /
    • 2000
  • 본 연구는 문서를 구성하고 있는 각 단락들(paragraphs)간의 단어공기정보(world co-occurrence)를 이용해 이들간의 관계를 바탕으로 중요단락을 추출하여 문서의 요약을 한다. 이같은 접근법 문서요약의 성능은 단락들간의 정보추출방법과 추출된 정보에 의한 중요단락 선택방법에 크게 좌우된다. 본 논문에서는 중요단락에 대한 선택을 할 때 기존의 방법론에서 발생하는 요약문의 가독성(readability)을 높이면서 또한 성능의 향상도 꾀할 수 있는 방법론을 제시한다.

  • PDF

Automatic Text Summarization Using Query Expansion (질의확장을 이용한 자동 문서요약)

  • 한경수;백대호;임해창
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2000.04b
    • /
    • pp.339-341
    • /
    • 2000
  • 문서요약이란 문서의 기본적인 내용을 유지하면서 문서의 복잡도를 줄이는 작업이다. 인터넷과 같은 정보기술의 발달로 정보의 양이 급증함에 따라, 정보 과적재(information over load) 문제의 해결을 위해 자동 문서요약시스템의 필요성이 대두되었다. 본 논문에서는 의사 적합성 피드백(pseudo relevance feedback)에 의한 질의확장(query expansion) 기법을 적용한 자동 문서요약 모델을 제안한다. 제안하는 모델의 특징은 질의를 분해함으로써, 적합성 피드백 과정에서 질의가 편향(bias)되어 요약이 잘못되는 문제를 방지할 수 있다는 것이다. 신문기사를 대상으로 평가한 결과 제안한 모델이 질의확장을 적용하지 않은 방법이나 하나의 질의만을 유지하는 일반적인 적합성 피드백 모델보다 더 좋은 성능을 보였다.

  • PDF

Automatic Text Summarization with Lexical Clustering (어휘 클러스터링을 이용한 자동 문서 요약)

  • 김건오;고영중;서정연
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2002.04b
    • /
    • pp.463-465
    • /
    • 2002
  • 자동 문서 요약 시스템은 문서내 담겨있는 정보를 최대만 표현하면서 문서의 크기를 줄이는 시스템이다. 본 논문에서는 어휘를 자동으로 클러스터링하여 문서 대표어를 찾고, 이를 제목과 조합하여 요약을 수행하는 시스템을 제안한다. 특히 이 시스템은 제목이 없는 문서도 요약을 수행할 수 있는 장점이 있다. 비교시스템으로는 제목, 위치, 빈도를 이용만 시스템을 구축하여 사용하였으며 30%, 10%, 그리고 4문장 요약에서 제안한 시스템은 모두 우수한 성능을 보였다.

  • PDF