• Journal of Korean Society of Industrial and Systems Engineering
• /
• v.47 no.2
• /
• pp.134-146
• /
• 2024

This study introduces and experimentally validates a novel approach that combines Instruction fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning to optimize the performance of Large Language Models (LLMs). These models have become revolutionary tools in natural language processing, showing remarkable performance across diverse application areas. However, optimizing their performance for specific domains necessitates fine-tuning of the base models (FMs), which is often limited by challenges such as data complexity and resource costs. The proposed approach aims to overcome these limitations by enhancing the performance of LLMs, particularly in the analysis precision and efficiency of national Research and Development (R&D) data. The study provides theoretical foundations and technical implementations of Instruction fine-tuning and LoRA fine-tuning. Through rigorous experimental validation, it is demonstrated that the proposed method significantly improves the precision and efficiency of data analysis, outperforming traditional fine-tuning methods. This enhancement is not only beneficial for national R&D data but also suggests potential applicability in various other data-centric domains, such as medical data analysis, financial forecasting, and educational assessments. The findings highlight the method's broad utility and significant contribution to advancing data analysis techniques in specialized knowledge domains, offering new possibilities for leveraging LLMs in complex and resource-intensive tasks. This research underscores the transformative potential of combining Instruction fine-tuning with LoRA fine-tuning to achieve superior performance in diverse applications, paving the way for more efficient and effective utilization of LLMs in both academic and industrial settings.

• Korean Journal of Childcare and Education
• /
• v.15 no.4
• /
• pp.115-134
• /
• 2019

Objective: This study was aimed to analyze the effects of activities using picture books for young children's language development and to identify factors that caused differences in these effects by applying meta-analysis. Methods: We conducted a homogeneity test of effect sizes on 21 Korean studies published in academic journals from 1990 to February 2018 and calculated the effect size by applying a random effect model. Additionally, we conducted a meta-ANOVA to investigate whether the effect sizes differed by types of language development, picture book activities, and environmental variables-such as place, time, and agent. Results: The results indicated that the effect sizes of the 21 studies were heterogeneous and the total effect size was 0.90, which was significantly large according to Cohen's standard. The effect sizes also varied by types of language development, picture book activities, and environmental variables. Conclusion/Implications: To increase the effects of activities using picture books for young children's language development, this study suggested the importance of picture book activities to be integrated with other play areas, teaching methods, and other print materials for the development of literacy abilities, and the link between home and early childhood education institutions.

• MALSORI
• /
• no.59
• /
• pp.53-68
• /
• 2006

We report the evaluation results of the Korean speech recognition platform called ECHOS. The platform has an object-oriented and reusable architecture so that researchers can easily evaluate their own algorithms. The platform has all intrinsic modules to build a large vocabulary speech recognizer: Noise reduction, end-point detection, feature extraction, hidden Markov model (HMM)-based acoustic modeling, cross-word modeling, n-gram language modeling, n-best search, word graph generation, and Korean-specific language processing. The platform supports both lexical search trees and finite-state networks. It performs word-dependent n-best search with bigram in the forward search stage, and rescores the lattice with trigram in the backward stage. In an 8000-word continuous speech recognition task, the platform with a lexical tree increases 40% of word errors but decreases 50% of recognition time compared to the HTK platform with flat lexicon. ECHOS reduces 40% of recognition errors through incorporation of cross-word modeling. With the number of Gaussian mixtures increasing to 16, it yields word accuracy comparable to the previous lexical tree-based platform, Julius.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2023.07a
• /
• pp.59-61
• /
• 2023

본 논문에서는 감정 분류 성능 향상을 위한 초거대 언어모델로부터의 추론 데이터셋 활용 방안을 제안한다. 이 방안은 Google Research의 'Chain of Thought'에서 영감을 받아 이를 적용하였으며, 추론 데이터는 ChatGPT와 같은 초거대 언어 모델로 생성하였다. 본 논문의 목표는 머신러닝 모델이 추론 데이터를 이해하고 적용하는 능력을 활용하여, 감정 분류 작업의 성능을 향상시키는 것이다. 초거대 언어 모델(ChatGPT)로부터 추출한 추론 데이터셋을 활용하여 감정 분류 모델을 훈련하였으며, 이 모델은 감정 분류 작업에서 향상된 성능을 보였다. 이를 통해 추론 데이터셋이 감정 분류에 있어서 큰 가치를 가질 수 있음을 증명하였다. 또한, 이 연구는 기존에 감정 분류 작업에 사용되던 데이터셋만을 활용한 모델과 비교하였을 때, 추론 데이터를 활용한 모델이 더 높은 성능을 보였음을 증명한다. 이 연구를 통해, 적은 비용으로 초거대 언어모델로부터 생성된 추론 데이터셋의 활용 가능성을 보여주고, 감정 분류 작업 성능을 향상시키는 새로운 방법을 제시한다. 제시한 방안은 감정 분류뿐만 아니라 다른 자연어처리 분야에서도 활용될 수 있으며, 더욱 정교한 자연어 이해와 처리가 가능함을 시사한다.

• The Journal of Bigdata
• /
• v.9 no.1
• /
• pp.75-88
• /
• 2024

Recently, there have been cases reported in the news of individuals experiencing symptoms of food poisoning after consuming raw beef purchased from online platforms, or reviews claiming that cherry tomatoes tasted bitter. This suggests the potential for analyzing food reviews on online platforms to detect food hazards, enabling government agencies, food manufacturers, and distributors to manage consumer food safety risks. This study proposes a classification model that uses sentiment analysis and large language models to analyze food reviews and detect negative ones, multi-labeling key food safety hazards (food poisoning, spoilage, chemical odors, foreign objects). The sentiment analysis model effectively minimized the misclassification of negative reviews with a low False Positive rate using a 'funnel' model. The multi-labeling model for food safety hazards showed high performance with both recall and accuracy over 96% when using GPT-4 Turbo compared to GPT-3.5. Government agencies, food manufacturers, and distributors can use the proposed model to monitor consumer reviews in real-time, detect potential food safety issues early, and manage risks. Such a system can protect corporate brand reputation, enhance consumer protection, and ultimately improve consumer health and safety.

• KIPS Transactions on Software and Data Engineering
• /
• v.12 no.8
• /
• pp.341-354
• /
• 2023

The title of a document is the brief summarization of the document. Readers can easily understand a document if we provide them with its title in their preferred styles and the languages. In this research, we propose a cross-lingual and style-based title generation model using multiple adapters. To train the model, we need a parallel corpus in several languages with different styles. It is quite difficult to construct this kind of parallel corpus; however, a monolingual title generation corpus of the same style can be built easily. Therefore, we apply a zero-shot strategy to generate a title in a different language and with a different style for an input document. A baseline model is Transformer consisting of an encoder and a decoder, pre-trained by several languages. The model is then equipped with multiple adapters for translation, languages, and styles. After the model learns a translation task from parallel corpus, it learns a title generation task from monolingual title generation corpus. When training the model with a task, we only activate an adapter that corresponds to the task. When generating a cross-lingual and style-based title, we only activate adapters that correspond to a target language and a target style. An experimental result shows that our proposed model is only as good as a pipeline model that first translates into a target language and then generates a title. There have been significant changes in natural language generation due to the emergence of large-scale language models. However, research to improve the performance of natural language generation using limited resources and limited data needs to continue. In this regard, this study seeks to explore the significance of such research.

• Journal of Korean Association for Spatial Structures
• /
• v.1 no.2 s.2
• /
• pp.59-66
• /
• 2001

This paper describes three modules for development of the Space Frame Integrated Design System(SFIDS). The Control Module is implemented to control the developed system. The Model Generation Module based on PATRAN user interface enables users to generate a complicated finite element model for space frame structures. The Optimum Design Module base on a branch of combinatorial optimization techniques which can realize the optimization of a structure having a large number of members designs optimum members of a space frame after evaluating analysis results. The Control Module and the Model Generation Module Is implemented by PATRAN Command Language(PCL) while C++ language is used in the Optimum Design Module. The core of the system is PATRAN database, in which the Model Generation Module creates information of a finite element model. Then, PATRAN creates Input files needed for the analysis program from the information of the finite element model in the database, and in turn, imports output results of analysis program to the database. Finally, the Optimum Design Module processes member grouping of a space frame based on the output results, and performs optimal member selection of a space frame. This process is repeated until the desired optimum structural members are obtained.

• KIISE Transactions on Computing Practices
• /
• v.22 no.7
• /
• pp.315-319
• /
• 2016

Recurrent neural network based language models (RNN LM) have shown improved results in language model researches. The RNN LMs are limited to post processing sessions, such as the N-best rescoring step of the wFST based speech recognition. However, it has considerable vocabulary problems that require large computing powers for the LM training. In this paper, we try to find the 1st pass N-gram model using word embedding, which is the simplified deep neural network. The class based language model (LM) can be a way to approach to this issue. We have built class based vocabulary through word embedding, by combining the class LM with word N-gram LM to evaluate the performance of LMs. In addition, we propose that part-of-speech (POS) tagging based LM shows an improvement of perplexity in all types of the LM tests.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.16 no.5
• /
• pp.279-291
• /
• 2023

In recent years, language model-based generative AI technologies have made remarkable progress. In particular, it has attracted a lot of attention due to its increasing potential in various fields such as summarization and code writing. As a reflection of this interest, the number of patent applications related to generative AI has been increasing rapidly. In order to understand these trends and develop strategies accordingly, future forecasting is key. Predictions can be used to better understand the future trends in the field of technology and develop more effective strategies. In this paper, we analyzed patents filed to date to identify the direction of development of language model-based generative AI. In particular, we took an in-depth look at research and invention activities in each country, focusing on application trends by year and detailed technology. Through this analysis, we tried to understand the detailed technologies contained in the core patents and predict the future development trends of generative AI.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.13 no.4
• /
• pp.306-311
• /
• 2020

This paper deals with how to improve the performance of Python language with various libraries when developing a model using big data. The Python language uses the Pandas library for processing spreadsheet-format data such as Excel. In processing data, Python operates on an in-memory basis. There is no performance issue when processing small scale of data. However, performance issues occur when processing large scale of data. Therefore, this paper introduces a method for distributed processing of execution tasks in a single cluster and multiple clusters by using a Dask library that can be used with Pandas when processing data. The experiment compares the speed of processing a simple exponential model using only Pandas on the same specification hardware and the speed of processing using a dask together. This paper presents a method to develop a model by distributing a large scale of data by CPU cores in terms of performance while maintaining that python's advantage of using various libraries is easy.