• Annual Conference on Human and Language Technology
• /
• 2021.10a
• /
• pp.335-339
• /
• 2021

기계독해(MRC)는 인공지능 알고리즘이 문서를 이해하고 질문에 대한 정답을 찾는 기술이다. MRC는 사전 학습 모델을 사용하여 높은 성능을 내고 있고, 일반 텍스트문서 뿐만 아니라 문서 내의 테이블(표)에서도 정답을 찾고자 하는 연구에 활발히 적용되고 있다. 본 연구에서는 기존의 사전학습 모델을 테이블 데이터에 활용하여 질의응답을 할 수 있는 방법을 제안한다. 더불어 테이블 데이터를 효율적으로 학습하기 위한 데이터 구성 방법을 소개한다. 사전학습 모델은 BERT[1]를 사용하여 테이블 정보를 인코딩하고 Masked Entity Recovery(MER) 방식을 사용한다. 테이블 질의응답 모델 학습을 위해 한국어 위키 문서에서 표와 연관 텍스트를 추출하여 사전학습을 진행하였고, 미세 조정은 샘플링한 테이블에 대한 질문-답변 데이터 약 7만건을 구성하여 진행하였다. 결과로 KorQuAD2.0 데이터셋의 테이블 관련 질문 데이터에서 EM 69.07, F1 78.34로 기존 연구보다 우수한 성능을 보였다.

• Annual Conference on Human and Language Technology
• /
• 2023.10a
• /
• pp.644-647
• /
• 2023

대화 시스템에서 대화 상태 추적은 사용자와의 대화를 진행하면서 사용자의 의도를 파악하여 시스템 응답을 결정하는데 있어서 중요한 역할을 수행한다. 특히 목적지향(task-oriented) 대화에서 사용자 목표(goal)를 만족시키기 위해서 대화 상태 추적은 필수적이다. 최근 다양한 자연어처리 다운스트림 태스크들이 사전학습 언어모델을 백본 네트워크로 사용하고 그 위에서 해당 도메인 태스크를 미세조정하는 방식으로 좋은 성능을 내고 있다. 본 논문에서는 한국어 토큰-프리(token-free) 사전학습 언어모델인 KeByT5B 사용하고 종단형(end-to-end) seq2seq 방식으로 미세조정을 수행한 한국어 생성 기반 대화 상태 추적 모델을 소개하고 관련하여 수행한 실험 결과를 설명한다.

• Annual Conference on Human and Language Technology
• /
• 2023.10a
• /
• pp.143-148
• /
• 2023

본 논문에서는 금융 도메인 특화 사전학습 언어모델인 KF-DeBERTa(Korean Finance DeBERTa)를 제안한다. KF-DeBERTa는 대규모의 금융 말뭉치를 기반으로 학습하였으며, Transformer 아키텍처와 DeBERTa의 특징을 기반으로 구성되었다. 범용 및 금융 도메인에 대한 평가에서 KF-DeBERTa는 기존 언어모델들에 비해 상당히 높은 성능을 보였다. 특히, 금융 도메인에서의 성능은 매우 두드러졌으며, 범용 도메인에서도 다른 모델들을 상회하는 성능을 나타냈다. KF-DeBERTa는 모델 크기 대비 높은 성능 효율성을 보여주었고, 앞으로 금융 도메인에서의 활용도가 기대된다.

• Journal of Information Processing Systems
• /
• v.18 no.5
• /
• pp.650-664
• /
• 2022

Mobile applications can be easily downloaded and installed via markets. However, malware and malicious applications containing unwanted advertisements exist in these application markets. Therefore, smartphone users install applications with reference to the application review to avoid such malicious applications. An application review typically comprises contents for evaluation; however, a false review with a specific purpose can be included. Such false reviews are known as fake reviews, and they can be generated using artificial intelligence (AI)-based text-generating models. Recently, AI-based text-generating models have been developed rapidly and demonstrate high-quality generated texts. Herein, we analyze the features of fake reviews generated from Generative Pre-Training-2 (GPT-2), an AI-based text-generating model and create a model to detect those fake reviews. First, we collect a real human-written application review from Kaggle. Subsequently, we identify features of the fake review using natural language processing and statistical analysis. Next, we generate fake review detection models using five types of machine-learning models trained using identified features. In terms of the performances of the fake review detection models, we achieved average F1-scores of 0.738, 0.723, and 0.730 for the fake review, real review, and overall classifications, respectively.

• Journal of Intelligence and Information Systems
• /
• v.28 no.2
• /
• pp.191-206
• /
• 2022

Recently, it is a de-facto approach to utilize a pre-trained language model(PLM) to achieve the state-of-the-art performance for various natural language tasks(called downstream tasks) such as sentiment analysis and question answering. However, similar to any other machine learning method, PLM tends to depend on the data distribution seen during the training phase and shows worse performance on the unseen (Out-of-Distribution) domain. Due to the aforementioned reason, there have been many efforts to develop domain-specified PLM for various fields such as medical and legal industries. In this paper, we discuss the training of a finance domain-specified PLM for the Korean language and its applications. Our finance domain-specified PLM, KB-BERT, is trained on a carefully curated financial corpus that includes domain-specific documents such as financial reports. We provide extensive performance evaluation results on three natural language tasks, topic classification, sentiment analysis, and question answering. Compared to the state-of-the-art Korean PLM models such as KoELECTRA and KLUE-RoBERTa, KB-BERT shows comparable performance on general datasets based on common corpora like Wikipedia and news articles. Moreover, KB-BERT outperforms compared models on finance domain datasets that require finance-specific knowledge to solve given problems.

• Journal of the Korea Convergence Society
• /
• v.13 no.1
• /
• pp.13-21
• /
• 2022

Recent deep learning-based research shows excellent performance in most natural language processing (NLP) fields with pre-trained language models. In particular, the auto-encoder-based language model proves its excellent performance and usefulness in various fields of Korean language understanding. However, the decoder-based Korean generative model even suffers from generating simple sentences. Also, there is few detailed research and data for the field of conversation where generative models are most commonly utilized. Therefore, this paper constructs multi-turn dialogue data for a Korean generative model. In addition, we compare and analyze the performance by improving the dialogue ability of the generative model through transfer learning. In addition, we propose a method of supplementing the insufficient dialogue generation ability of the model by extracting recommended response candidates from external knowledge information through a retrival model.

• Journal of Internet Computing and Services
• /
• v.25 no.1
• /
• pp.57-67
• /
• 2024

This research investigates training methods for large language models to accurately identify sentiments and comprehend information about increasing and decreasing fluctuations in the financial domain. The main goal is to identify suitable datasets that enable these models to effectively understand expressions related to financial increases and decreases. For this purpose, we selected sentences from Wall Street Journal that included relevant financial terms and sentences generated by GPT-3.5-turbo-1106 for post-training. We assessed the impact of these datasets on language model performance using Financial PhraseBank, a benchmark dataset for financial sentiment analysis. Our findings demonstrate that post-training FinBERT, a model specialized in finance, outperformed the similarly post-trained BERT, a general domain model. Moreover, post-training with actual financial news proved to be more effective than using generated sentences, though in scenarios requiring higher generalization, models trained on generated sentences performed better. This suggests that aligning the model's domain with the domain of the area intended for improvement and choosing the right dataset are crucial for enhancing a language model's understanding and sentiment prediction accuracy. These results offer a methodology for optimizing language model performance in financial sentiment analysis tasks and suggest future research directions for more nuanced language understanding and sentiment analysis in finance. This research provides valuable insights not only for the financial sector but also for language model training across various domains.

• Journal of the Korean Society of Safety
• /
• v.38 no.5
• /
• pp.51-57
• /
• 2023

Numerous studies have been conducted to analyze the causal relationships of maritime accidents using natural language processing techniques. However, when multiple causes and effects are associated with a single accident, the effectiveness of extracting these causal relations diminishes. To address this challenge, we compiled a dataset using verdicts from maritime accident cases in this study, analyzed their causal relations, and applied labeling considering the association information of various causes and effects. In addition, to validate the efficacy of our proposed methodology, we fine-tuned the KoELECTRA Korean language model. The results of our validation process demonstrated the ability of our approach to successfully extract multiple causal relationships from maritime accident cases.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.16 no.4
• /
• pp.193-202
• /
• 2023

In recent years, the outstanding performance of large language models (LLMs) trained on extensive datasets has become a hot topic. Since studies on LLMs are available on open-source approaches, the ecosystem is expanding rapidly. Models that are task-specific, lightweight, and high-performing are being actively disseminated using additional training techniques using pre-trained LLMs as foundation models. On the other hand, the performance of LLMs for Korean is subpar because English comprises a significant proportion of the training dataset of existing LLMs. Therefore, research is being carried out on Korean-specific LLMs that allow for further learning with Korean language data. This paper identifies trends of open source based LLMs and introduces research on Korean specific large language models; moreover, the applications and limitations of large language models are described.

• KIPS Transactions on Software and Data Engineering
• /
• v.12 no.8
• /
• pp.365-370
• /
• 2023

This paper aims to compare the performance of speech data classification into two groups, adult and elderly, based on the acoustic and linguistic characteristics that change due to aging, such as changes in respiratory patterns, phonation, pitch, frequency, and language expression ability. For acoustic features we used attributes related to the frequency, amplitude, and spectrum of speech voices. As for linguistic features, we extracted hidden state vector representations containing contextual information from the transcription of speech utterances using KoBERT, a Korean pre-trained language model that has shown excellent performance in natural language processing tasks. The classification performance of each model trained based on acoustic and linguistic features was evaluated, and the F1 scores of each model for the two classes, adult and elderly, were examined after address the class imbalance problem by down-sampling. The experimental results showed that using linguistic features provided better performance for classifying adult and elderly than using acoustic features, and even when the class proportions were equal, the classification performance for adult was higher than that for elderly.