• Title/Summary/Keyword: parallel corpus

Search Result 66, Processing Time 0.028 seconds

Utilizing Local Bilingual Embeddings on Korean-English Law Data (한국어-영어 법률 말뭉치의 로컬 이중 언어 임베딩)

  • Choi, Soon-Young;Matteson, Andrew Stuart;Lim, Heui-Seok
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.10
    • /
    • pp.45-53
    • /
    • 2018
  • Recently, studies about bilingual word embedding have been gaining much attention. However, bilingual word embedding with Korean is not actively pursued due to the difficulty in obtaining a sizable, high quality corpus. Local embeddings that can be applied to specific domains are relatively rare. Additionally, multi-word vocabulary is problematic due to the lack of one-to-one word-level correspondence in translation pairs. In this paper, we crawl 868,163 paragraphs from a Korean-English law corpus and propose three mapping strategies for word embedding. These strategies address the aforementioned issues including multi-word translation and improve translation pair quality on paragraph-aligned data. We demonstrate a twofold increase in translation pair quality compared to the global bilingual word embedding baseline.

PCMM-Based Feature Compensation Method Using Multiple Model to Cope with Time-Varying Noise (시변 잡음에 대처하기 위한 다중 모델을 이용한 PCMM 기반 특징 보상 기법)

  • 김우일;고한석
    • The Journal of the Acoustical Society of Korea
    • /
    • v.23 no.6
    • /
    • pp.473-480
    • /
    • 2004
  • In this paper we propose an effective feature compensation scheme based on the speech model in order to achieve robust speech recognition. The proposed feature compensation method is based on parallel combined mixture model (PCMM). The previous PCMM works require a highly sophisticated procedure for estimation of the combined mixture model in order to reflect the time-varying noisy conditions at every utterance. The proposed schemes can cope with the time-varying background noise by employing the interpolation method of the multiple mixture models. We apply the‘data-driven’method to PCMM tot move reliable model combination and introduce a frame-synched version for estimation of environments posteriori. In order to reduce the computational complexity due to multiple models, we propose a technique for mixture sharing. The statistically similar Gaussian components are selected and the smoothed versions are generated for sharing. The performance is examined over Aurora 2.0 and speech corpus recorded while car-driving. The experimental results indicate that the proposed schemes are effective in realizing robust speech recognition and reducing the computational complexities under both simulated environments and real-life conditions.

Term Clustering and Duplicate Distribution for Efficient Parallel Information Retrieval (효율적인 병렬정보검색을 위한 색인어 군집화 및 분산저장 기법)

  • 강재호;양재완;정성원;류광렬;권혁철;정상화
    • Journal of KIISE:Software and Applications
    • /
    • v.30 no.1_2
    • /
    • pp.129-139
    • /
    • 2003
  • The PC cluster architecture is considered as a cost-effective alternative to the existing supercomputers for realizing a high-performance information retrieval (IR) system. To implement an efficient IR system on a PC cluster, it is essential to achieve maximum parallelism by having the data appropriately distributed to the local hard disks of the PCs in such a way that the disk I/O and the subsequent computation are distributed as evenly as possible to all the PCs. If the terms in the inverted index file can be classified to closely related clusters, the parallelism can be maximized by distributing them to the PCs in an interleaved manner. One of the goals of this research is the development of methods for automatically clustering the terms based on the likelihood of the terms' co-occurrence in the same query. Also, in this paper, we propose a method for duplicate distribution of inverted index records among the PCs to achieve fault-tolerance as well as dynamic load balancing. Experiments with a large corpus revealed the efficiency and effectiveness of our method.

Expression of Luteinizing Hormone (LH) Gene in Rat Uterus and Epididymis (흰쥐 자궁과 부정소에서의 Luteinizing Hormone (LH) 유전자 발현)

  • Lee, Sung-Ho;Lee, Young-Ki
    • Clinical and Experimental Reproductive Medicine
    • /
    • v.26 no.2
    • /
    • pp.157-161
    • /
    • 1999
  • Recent studies clearly demonstrated that the novel expression of LH gene in the rat testis, and suggested the local action of the LH-like molecule. The present study was performed to analyze the expression of LH genes in the rat accessory reproductive organs. Expression of LH subunit genes in the rat uterus and epididymis was demonstrated by reverse transcription-polymerase chain reaction (RT-PCR) and specific LH radioimmunoassay (RIA). The $LH_{beta}$ transcripts in these organs contained the published cDNA structure, the pituitary type exons 1-3, which encoded the entire $LH_{beta}$ polypeptide. Presence of the transcripts for the ${\alpha}$-subunit in the rat reproductive tissues were also confirmed by RT-PCR. In the LH RIA, significant levels of LH were detected in crude extracts from the rat ovary, uterus and epididymis. The competition curves with increasing amount of tissue extracts were parallel with those of standard peptide, indicating that the immunoreactive LH-like materials in these tissues are similar to authentic pituitary LH molecule. In rat epididymis, the highest amount of immunoreactive LH was detected in corpus area. Our findings demonstrated that the genes for LH subunits are expressed in the rat accessory reproductive organs, and suggested that these extrapituitary LH may act as a local regulator with auto and/or paracrine manner.

  • PDF

Factors Regulating Changes of Head-to-Head Agglutinability in Boar Spermatozoa During Epididymal Transit and Capacitation In Vitro - Review-

  • Hiroshi, Harayama;Seishiro, Kato
    • Asian-Australasian Journal of Animal Sciences
    • /
    • v.14 no.8
    • /
    • pp.1196-1202
    • /
    • 2001
  • In boar spermatozoa, the head-to-head agglutinability changes in parallel with the development of the fertilizing ability. Namely, both abilities gradually increase in the distal caput and corpus epididymides, but are subsequently suppressed in the cauda epididymidis. It has been postulated that these changes of the agglutinability are controlled via sperm interaction with specific epididymal plasma factors including agglutination mediators (agglutinins) and inhibitors (anti-agglutinins). Expression of these abilities (sperm agglutination and capacitation) is hardly observed in spermatozoa immediately. after ejaculation, but it occurs during incubation in a capacitation medium. Recently, we have purified and characterized epididymal plasma anti-agglutinin for boar spermatozoa. Moreover, we have conducted a series of experiments to reveal biological significance and mechanism of the head-to-head agglutination and have accumulated data indicating that boar sperm agglutination is mediated by capacitation-supporting factors including calcium, bicarbonate and sterol acceptors. This review introduces our recent data and discusses a possible mechanism for suppression of the agglutinability in the distal epididymidis and relationship between agglutinability and fertilizing ability.

Construction of a Parallel Corpus for Instant Messenger Spelling Correction and Related Issues (메신저 맞춤법 교정 병렬 말뭉치의 구축과 쟁점)

  • HUANG YINXIA;Jin-san An;Kil-im Nam
    • Annual Conference on Human and Language Technology
    • /
    • 2022.10a
    • /
    • pp.545-550
    • /
    • 2022
  • 본 연구의 목적은 2021년 메신저 언어 200만 어절을 대상으로 수행된 맞춤법 교정 병렬 말뭉치의 설계와 구축의 쟁점을 소개하고, 교정 말뭉치의 주요 교정 및 주석 내용을 기술함으로써 맞춤법 교정 병렬 말뭉치의 특성을 분석하는 것이다. 2021년 맞춤법 교정 병렬 말뭉치의 주요 목표는 메신저 언어의 특수성을 살림과 동시에 형태소 분석이나 기계 번역 등 한국어 처리 도구가 분석할 수 있는 수준으로 교정하는 다소 상충되는 목적을 구현하는 것이었는데, 이는 교정의 수준과 병렬의 단위 설정 등 상당한 쟁점을 내포한다. 본 연구에서는 말뭉치 구축 시점에서 미처 논의하지 못한 교정 수준의 쟁점과 교정 전후의 통계적 특성을 함께 논의하고자 하며, 다음과 같은 몇 가지 하위 내용을 중심으로 논의하고자 한다.첫째, 맞춤법 교정 병렬 말뭉치의 구조 설계와 구축 절차에 대한 논의로, 2022년 초 국내 최초로 공개된 한국어 맞춤법 교정 병렬 말뭉치('모두의 말뭉치'의 일부)의 구축 과정에서 논의되어 온 말뭉치 구조 설계와 구축 절차를 논의한다. 둘째, 문장 단위로 정렬된 맞춤법 교정 말뭉치에서 관찰 가능한 띄어쓰기, 미등재어, 부호형 이모티콘 등의 메신저 언어의 몇 가지 특성을 살펴본다. 마지막으로, 2021년 메신저 맞춤법 교정 말뭉치의 구축 단계에서 미처 논의되지 못한 남은 문제들을 각각 데이터 구조 설계와 구축 차원의 주요 쟁점을 중심으로 논의한다. 특히 메신저 맞춤법 병렬 말뭉치의 주요 목표인 사전학습 언어모델의 학습데이터로서의 가치와 메신저 언어 연구의 기반 자료 구축의 관점에서 맞춤법 교정 병렬 말뭉치 구축의 의의와 향후 과제를 논의하고자 한다.

  • PDF

Parallel Corpus Filtering and Korean-Optimized Subword Tokenization for Machine Translation (병렬 코퍼스 필터링과 한국어에 최적화된 서브 워드 분절 기법을 이용한 기계번역)

  • Park, Chanjun;kim, Gyeongmin;Lim, Heuiseok
    • Annual Conference on Human and Language Technology
    • /
    • 2019.10a
    • /
    • pp.221-224
    • /
    • 2019
  • 딥러닝을 이용한 Neural Machine Translation(NMT)의 등장으로 기계번역 분야에서 기존의 규칙 기반,통계기반 방식을 압도하는 좋은 성능을 보이고 있다. 본 논문은 기계번역 모델도 중요하지만 무엇보다 중요한 것은 고품질의 학습데이터를 구성하는 일과 전처리라고 판단하여 이에 관련된 다양한 실험을 진행하였다. 인공신경망 기계번역 시스템의 학습데이터 즉 병렬 코퍼스를 구축할 때 양질의 데이터를 확보하는 것이 무엇보다 중요하다. 그러나 양질의 데이터를 구하는 일은 저작권 확보의 문제, 병렬 말뭉치 구축의 어려움, 노이즈 등을 이유로 쉽지 않은 상황이다. 본 논문은 고품질의 학습데이터를 구축하기 위하여 병렬 코퍼스 필터링 기법을 제시한다. 병렬 코퍼스 필터링이란 정제와 다르게 학습 데이터에 부합하지 않다고 판단되며 소스, 타겟 쌍을 함께 삭제 시켜 버린다. 또한 기계번역에서 무엇보다 중요한 단계는 바로 Subword Tokenization 단계이다. 본 논문은 다양한 실험을 통하여 한-영 기계번역에서 가장 높은 성능을 보이는 Subword Tokenization 방법론을 제시한다. 오픈 된 한-영 병렬 말뭉치로 실험을 진행한 결과 병렬 코퍼스 필터링을 진행한 데이터로 만든 모델이 더 좋은 BLEU 점수를 보였으며 본 논문에서 제안하는 형태소 분석 단위 분리를 진행 후 Unigram이 반영된 SentencePiece 모델로 Subword Tokenization를 진행 하였을 시 가장 좋은 성능을 보였다.

  • PDF

Addressing Low-Resource Problems in Statistical Machine Translation of Manual Signals in Sign Language (말뭉치 자원 희소성에 따른 통계적 수지 신호 번역 문제의 해결)

  • Park, Hancheol;Kim, Jung-Ho;Park, Jong C.
    • Journal of KIISE
    • /
    • v.44 no.2
    • /
    • pp.163-170
    • /
    • 2017
  • Despite the rise of studies in spoken to sign language translation, low-resource problems of sign language corpus have been rarely addressed. As a first step towards translating from spoken to sign language, we addressed the problems arising from resource scarcity when translating spoken language to manual signals translation using statistical machine translation techniques. More specifically, we proposed three preprocessing methods: 1) paraphrase generation, which increases the size of the corpora, 2) lemmatization, which increases the frequency of each word in the corpora and the translatability of new input words in spoken language, and 3) elimination of function words that are not glossed into manual signals, which match the corresponding constituents of the bilingual sentence pairs. In our experiments, we used different types of English-American sign language parallel corpora. The experimental results showed that the system with each method and the combination of the methods improved the quality of manual signals translation, regardless of the type of the corpora.

Enhancing Performance of Bilingual Lexicon Extraction through Refinement of Pivot-Context Vectors (중간언어 문맥벡터의 정제를 통한 이중언어 사전 구축의 성능개선)

  • Kwon, Hong-Seok;Seo, Hyung-Won;Kim, Jae-Hoon
    • Journal of KIISE:Software and Applications
    • /
    • v.41 no.7
    • /
    • pp.492-500
    • /
    • 2014
  • This paper presents the performance enhancement of automatic bilingual lexicon extraction by using refinement of pivot-context vectors under the standard pivot-based approach, which is very effective method for less-resource language pairs. In this paper, we gradually improve the performance through two different refinements of pivot-context vectors: One is to filter out unhelpful elements of the pivot-context vectors and to revise the values of the vectors through bidirectional translation probabilities estimated by Anymalign and another one is to remove non-noun elements from the original vectors. In this paper, experiments have been conducted on two different language pairs that are bi-directional Korean-Spanish and Korean-French, respectively. The experimental results have demonstrated that our method for high-frequency words shows at least 48.5% at the top 1 and up to 88.5% at the top 20 and for the low-frequency words at least 43.3% at the top 1 and up to 48.9% at the top 20.

Knowledge-poor Term Translation using Common Base Axis with application to Korean-English Cross-Language Information Retrieval (과도한 지식을 요구하지 않는 공통기반축에 의한 용어 번역과 한영 교차정보검색에의 응용)

  • 최용석;최기선
    • Korean Journal of Cognitive Science
    • /
    • v.14 no.1
    • /
    • pp.29-40
    • /
    • 2003
  • Cross-Language Information Retrieval (CLIR) deals with the documents in various languages by one language query. A user who uses one language can retrieve the documents in another language through CLIR system. In CLIR, query translation method is known to be more efficient. For the better performance of query translation, we need more resources like dictionary, ontology, and parallel/comparable corpus but usually not available. This paper proposes a new concept called the Common Base Axis which is adapted to Korean-English Query translation ann a new weighting method in dictionary based query translation. The essential idea is that we can express Korean and English word in one vector space by Common Base Axis and use it in calculating sense distance for query weighting. The experiments show that Common Base Axis gives us good performance without ontology and is especially good for one word query translation.

  • PDF