• Title/Summary/Keyword: sentence similarity

Search Result 82, Processing Time 0.026 seconds

A Korean Text Summarization System Using Aggregate Similarity (도합유사도를 이용한 한국어 문서요약 시스템)

  • 김재훈;김준홍
    • Korean Journal of Cognitive Science
    • /
    • v.12 no.1_2
    • /
    • pp.35-42
    • /
    • 2001
  • In this paper. a document is represented as a weighted graph called a text relationship map. In the graph. a node represents a vector of nouns in a sentence, an edge completely connects other nodes. and a weight on the edge is a value of the similarity between two nodes. The similarity is based on the word overlap between the corresponding nodes. The importance of a node. called an aggregate similarity in this paper. is defined as the sum of weights on the links connecting it to other nodes on the map. In this paper. we present a Korean text summarization system using the aggregate similarity. To evaluate our system, we used two test collection, one collection (PAPER-InCon) consists of 100 papers in the field of computer science: the other collection (NEWS) is composed of 105 articles in the newspapers and had built by KOROlC. Under the compression rate of 20%. we achieved the recall of 46.6% (PAPER-InCon) and 30.5% (NEWS) and the precision of 76.9% (PAPER-InCon) and 42.3% (NEWS).

  • PDF

Summarizing the Differences in Chinese-Vietnamese Bilingual News

  • Wu, Jinjuan;Yu, Zhengtao;Liu, Shulong;Zhang, Yafei;Gao, Shengxiang
    • Journal of Information Processing Systems
    • /
    • v.15 no.6
    • /
    • pp.1365-1377
    • /
    • 2019
  • Summarizing the differences in Chinese-Vietnamese bilingual news plays an important supporting role in the comparative analysis of news views between China and Vietnam. Aiming at cross-language problems in the analysis of the differences between Chinese and Vietnamese bilingual news, we propose a new method of summarizing the differences based on an undirected graph model. The method extracts elements to represent the sentences, and builds a bridge between different languages based on Wikipedia's multilingual concept description page. Firstly, we calculate the similarity between Chinese and Vietnamese news sentences, and filter the bilingual sentences accordingly. Then we use the filtered sentences as nodes and the similarity grade as the weight of the edge to construct an undirected graph model. Finally, combining the random walk algorithm, the weight of the node is calculated according to the weight of the edge, and sentences with highest weight can be extracted as the difference summary. The experiment results show that our proposed approach achieved the highest score of 0.1837 on the annotated test set, which outperforms the state-of-the-art summarization models.

An Effective Sentence Similarity Measure Method Based FAQ System Using Self-Attentive Sentence Embedding (Self-Attention 기반의 문장 임베딩을 이용한 효과적인 문장 유사도 기법 기반의 FAQ 시스템)

  • Kim, Bosung;Kim, Juae;Lee, Jeong-Eom;Kim, Seona;Ko, Youngjoong;Seo, Jungyun
    • Annual Conference on Human and Language Technology
    • /
    • 2018.10a
    • /
    • pp.361-363
    • /
    • 2018
  • FAQ 시스템은 주어진 질문과 가장 유사한 질의를 찾아 이에 대한 답을 제공하는 시스템이다. 질의 간의 유사도를 측정하기 위해 문장을 벡터로 표현하며 일반적으로 TFIDF, Okapi BM25와 같은 방법으로 계산한 단어 가중치 벡터를 이용하여 문장을 표현한다. 하지만 단어 가중치 벡터는 어휘적 정보를 표현하는데 유용한 반면 단어의 의미적인(semantic) 정보는 표현하기 어렵다. 본 논문에서는 이를 보완하고자 딥러닝을 이용한 문장 임베딩을 구축하고 단어 가중치 벡터와 문장 임베딩을 조합한 문장 유사도 계산 모델을 제안한다. 또한 문장 임베딩 구현 시 self-attention 기법을 적용하여 문장 내 중요한 부분에 가중치를 주었다. 실험 결과 제안하는 유사도 계산 모델은 비교 모델에 비해 모두 높은 성능을 보였고 self-attention을 적용한 실험에서는 추가적인 성능 향상이 있었다.

  • PDF

An Analysis on the Problem Solving of Korean and American 3rd Grade Students in the Addition and Subtraction with Natural Numbers (한국과 미국 초등학교 3학년 학생들의 자연수 덧셈과 뺄셈 문제해결 분석)

  • Lee, Dae Hyun
    • Education of Primary School Mathematics
    • /
    • v.19 no.3
    • /
    • pp.177-191
    • /
    • 2016
  • Students can calculate the addition and subtraction problem using informal knowledge before receiving the formal instruction. Recently, the value that a computation lesson focus on the understanding and developing the various strategies is highlighted by curriculum developers as well as in reports. Ideally, a educational setting and classroom culture reflected students' learning and problem solving strategies. So, this paper analyzed the similarity and difference with respect to the numeric sentence and word problem in the addition and subtraction. The subjects for the study were 100 third-grade Korean students and 68 third-grade American students. Researcher developed the questionnaire in the addition and subtraction and used it for the survey. The following results have been drawn from this study. The computational ability of Korean students was higher than that of American students in both the numeric sentence and word problem. And it was revealed the differences of the strategies which were used problem solving process. Korean students tended to use algorithms and numbers' characters and relations, but American students tended to use the drawings and algorithms with drawings.

Building a Korean-English Parallel Corpus by Measuring Sentence Similarities Using Sequential Matching of Language Resources and Topic Modeling (언어 자원과 토픽 모델의 순차 매칭을 이용한 유사 문장 계산 기반의 위키피디아 한국어-영어 병렬 말뭉치 구축)

  • Cheon, JuRyong;Ko, YoungJoong
    • Journal of KIISE
    • /
    • v.42 no.7
    • /
    • pp.901-909
    • /
    • 2015
  • In this paper, to build a parallel corpus between Korean and English in Wikipedia. We proposed a method to find similar sentences based on language resources and topic modeling. We first applied language resources(Wiki-dictionary, numbers, and online dictionary in Daum) to match word sequentially. We construct the Wiki-dictionary using titles in Wikipedia. In order to take advantages of the Wikipedia, we used translation probability in the Wiki-dictionary for word matching. In addition, we improved the accuracy of sentence similarity measuring method by using word distribution based on topic modeling. In the experiment, a previous study showed 48.4% of F1-score with only language resources based on linear combination and 51.6% with the topic modeling considering entire word distributions additionally. However, our proposed methods with sequential matching added translation probability to language resources and achieved 9.9% (58.3%) better result than the previous study. When using the proposed sequential matching method of language resources and topic modeling after considering important word distributions, the proposed system achieved 7.5%(59.1%) better than the previous study.

Probing Sentence Embeddings in L2 Learners' LSTM Neural Language Models Using Adaptation Learning

  • Kim, Euhee
    • Journal of the Korea Society of Computer and Information
    • /
    • v.27 no.3
    • /
    • pp.13-23
    • /
    • 2022
  • In this study we leveraged a probing method to evaluate how a pre-trained L2 LSTM language model represents sentences with relative and coordinate clauses. The probing experiment employed adapted models based on the pre-trained L2 language models to trace the syntactic properties of sentence embedding vector representations. The dataset for probing was automatically generated using several templates related to different sentence structures. To classify the syntactic properties of sentences for each probing task, we measured the adaptation effects of the language models using syntactic priming. We performed linear mixed-effects model analyses to analyze the relation between adaptation effects in a complex statistical manner and reveal how the L2 language models represent syntactic features for English sentences. When the L2 language models were compared with the baseline L1 Gulordava language models, the analogous results were found for each probing task. In addition, it was confirmed that the L2 language models contain syntactic features of relative and coordinate clauses hierarchically in the sentence embedding representations.

A Framework to Evaluate Communication Quality of Operators in Nuclear Power Plants Using Cosine Similarity (코사인 유사도를 이용한 원자력발전소 운전원 커뮤니케이션 품질 평가 프레임워크)

  • Kim, Seung-Hwan;Park, Jin-Kyun;Han, Sang-Yong
    • Journal of the Korea Society of Computer and Information
    • /
    • v.15 no.9
    • /
    • pp.165-172
    • /
    • 2010
  • Communication problems have been regarded as one of the biggest causes in trouble in many industries. This led to extensive research on communication as a part of human error analysis. The results of existing researches have revealed that maintaining a good quality of communication is essential to secure the safety of a large and complex process system. In this paper, we suggested a method to measure the quality of communication during off-normal situation in main control room of nuclear power plants. It evaluates the cosine similarity that is a measure of sentence similarity between two operators by finding the cosine of the angle between them. To check the applicability of the method to evaluate communication quality, we compared the result of communication quality analysis with the result of operation performance that was performed by operators under simulated environment.

Music Structure Analysis and Application (악곡구조 분석과 활용)

  • Seo, Jung-Bum;Bae, Jae-Hak
    • The KIPS Transactions:PartB
    • /
    • v.14B no.1 s.111
    • /
    • pp.33-42
    • /
    • 2007
  • This paper presents a new methodology for music structure analysis which facilitates rhetoric-based music summarization. Similarity analysis of musical constituents suggests the structure of a musical piece. We can recognize its musical form from the structure. Musical forms have rhetorical characteristics of their on. We have utilized the characteristics for locating musical motifs. Motif extraction is to music summarization what topic sentence extraction is to text summarization. We have evaluated the effectiveness of this methodology through a popular music case study.

Topic-based Multi-document Summarization Using Non-negative Matrix Factorization and K-means (비음수 행렬 분해와 K-means를 이용한 주제기반의 다중문서요약)

  • Park, Sun;Lee, Ju-Hong
    • Journal of KIISE:Software and Applications
    • /
    • v.35 no.4
    • /
    • pp.255-264
    • /
    • 2008
  • This paper proposes a novel method using K-means and Non-negative matrix factorization (NMF) for topic -based multi-document summarization. NMF decomposes weighted term by sentence matrix into two sparse non-negative matrices: semantic feature matrix and semantic variable matrix. Obtained semantic features are comprehensible intuitively. Weighted similarity between topic and semantic features can prevent meaningless sentences that are similar to a topic from being selected. K-means clustering removes noises from sentences so that biased semantics of documents are not reflected to summaries. Besides, coherence of document summaries can be enhanced by arranging selected sentences in the order of their ranks. The experimental results show that the proposed method achieves better performance than other methods.

SimKoR: A Sentence Similarity Dataset based on Korean Review Data and Its Application to Contrastive Learning for NLP (SimKoR: 한국어 리뷰 데이터를 활용한 문장 유사도 데이터셋 제안 및 대조학습에서의 활용 방안 )

  • Jaemin Kim;Yohan Na;Kangmin Kim;Sang Rak Lee;Dong-Kyu Chae
    • Annual Conference on Human and Language Technology
    • /
    • 2022.10a
    • /
    • pp.245-248
    • /
    • 2022
  • 최근 자연어 처리 분야에서 문맥적 의미를 반영하기 위한 대조학습 (contrastive learning) 에 대한 연구가 활발히 이뤄지고 있다. 이 때 대조학습을 위한 양질의 학습 (training) 데이터와 검증 (validation) 데이터를 이용하는 것이 중요하다. 그러나 한국어의 경우 대다수의 데이터셋이 영어로 된 데이터를 한국어로 기계 번역하여 검토 후 제공되는 데이터셋 밖에 존재하지 않는다. 이는 기계번역의 성능에 의존하는 단점을 갖고 있다. 본 논문에서는 한국어 리뷰 데이터로 임베딩의 의미 반영 정도를 측정할 수 있는 간단한 검증 데이터셋 구축 방법을 제안하고, 이를 활용한 데이터셋인 SimKoR (Similarity Korean Review dataset) 을 제안한다. 제안하는 검증 데이터셋을 이용해서 대조학습을 수행하고 효과성을 보인다.

  • PDF