• The Journal of Korean Institute of Next Generation Computing
• /
• v.13 no.5
• /
• pp.124-130
• /
• 2017

Character encoding is the method used to represent characters or symbols on a computer, and there are many encoding detection software tools. For the widely used encoding detection software"uchardet", the accuracy of encoding detection of unmodified normal text document is 91.39%, but the accuracy of language detection is only 32.09%. Also, if a text document is encrypted by substitution, the accuracy of encoding detection is 3.55% and the accuracy of language detection is 0.06%. Therefore, in this paper, we propose encoding and language detection of text document using the deep learning algorithm called LSTM(Long Short-Term Memory). The results of LSTM are better than encoding detection software"uchardet". The accuracy of encoding detection of normal text document using the LSTM is 99.89% and the accuracy of language detection is 99.92%. Also, if a text document is encrypted by substitution, the accuracy of encoding detection is 99.26%, the accuracy of language detection is 99.77%.

• Proceedings of the Korean Society for Language and Information Conference
• /
• 2007.11a
• /
• pp.515-521
• /
• 2007

The text categorization is an important field for the automatic text information processing. Moreover, the authorship identification of a text can be treated as a special text categorization. This paper adopts the conceptual primitives' expression based on the Hierarchical Network of Concepts (HNC) theory, which can describe the words meaning in hierarchical symbols, in order to avoid the sparse data shortcoming that is aroused by the natural language surface features in text categorization. The KNN algorithm is used as computing classification element. Then, the experiment has been done on the Chinese text authorship identification. The experiment result gives out that the processing mode that is put forward in this paper achieves high correct rate, so it is feasible for the text authorship identification.

• Proceedings of the Korean Society for Language and Information Conference
• /
• 2007.11a
• /
• pp.192-201
• /
• 2007

This paper is a study on constructing a natural language interface to database, concentrating on generating textual answers. TGEN, a system that generates textual answer from query result tables is presented. The TGEN architecture guarantees its portability across domains. A combination of a frame-based approach and natural language generation techniques in the TGEN provides text fluency and text flexibility. The implementation result shows that this approach is feasible while a deep NLG approach is still far to be reached.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2002.04b
• /
• pp.1013-1016
• /
• 2002

제어 시스템에 사용되는 흐름 제어 언어로는 IL(Instruction List), ST(Structured Text), FBD(Function Block Diagram), SFC(Sequential Function Chart), LD (Ladder Diagram)가 있다. 일반적으로 제어 시스템에 탑재하여 사용하는 언어는 상기 언어 중 두 종류 이하의 특정 언어로 제한되어 있다. 이러한 제약을 보완하기 위해, 모든 흐름 제어 언어를 통합 분석할 수 있는 통합 분석기가 필요하다. 본 논문에서는 흐름 제어 언어의 통합 분석 처리가 가능하도록 그래픽 표현의 FBD 를 문자 표현의 EST(Extended Structured Text)로 변형하는 규칙과 문자 표현의 EST 를 IL 로 변형하는 규칙을 제시한다. 언어간의 변형 과정에서 FBD 를 ST 로 표현할 수 없는 부분을 EST 로 정의한다. 또한 본 논문에서 제안된 EST 를 기반으로 통합 분석기의 구조를 제시한다.

• Journal of The Korean Association of Information Education
• /
• v.23 no.4
• /
• pp.355-362
• /
• 2019

Elementary students are studying software training to teach coding education using text-based languages such as Python. In general, these higher-level languages support learning activities in combination with a kits for physical computing or various programming languages, in contrast to block-coding programming languages. In this study, we conducted a coding project based on computational thinking using the ARCS model to overcome the difficulties of text-based language. The results of the experiment show that students are generally confident and interested in programming. Especially, the understanding of repetition, function, and object was high in the change of computational thinking power, so this trend is believed to be due to the use of text-based languages and the Python module.

• Journal of the Korea Society of Computer and Information
• /
• v.25 no.5
• /
• pp.187-197
• /
• 2020

Recently, various methods of text embedding using deep learning algorithms have been proposed. Especially, the way of using pre-trained language model which uses tremendous amount of text data in training is mainly applied for embedding new text data. However, traditional pre-trained language model has some limitations that it is hard to understand unique context of new text data when the text has too many tokens. In this paper, we propose self-supervised learning-based fine tuning method for pre-trained language model to infer vectors of long-text. Also, we applied our method to news articles and classified them into categories and compared classification accuracy with traditional models. As a result, it was confirmed that the vector generated by the proposed model more accurately expresses the inherent characteristics of the document than the vectors generated by the traditional models.

• Annual Conference on Human and Language Technology
• /
• 2021.10a
• /
• pp.612-616
• /
• 2021

자연어 처리의 다양한 task 들을 잘 수행하기 위해서 텍스트 내에서 적절한 용어를 골라내는 것은 중요하다. 텍스트에서 적절한 용어들을 자동으로 추출하기 위해 다양한 모델들을 학습시켜 용어의 특성을 잘 반영하는 n 그램을 추출할 수 있다. 본 연구에서는 기존에 존재하는 신경망 모델들을 조합하여 자동 용어 추출 성능을 개선할 수 있는 방법들을 제시하고 각각의 결과들을 비교한다.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.8 no.1
• /
• pp.53-62
• /
• 2008

Cross-lingual text retrieval(CLTR) is the information retrieval in which a user tries to search a set of documents written in one language for a query another language. This thesis proposes a CLTR system based on fuzzy multilingual thesaurus to handle a partial matching between terms of two different languages. The proposed CLTR system uses a fuzzy term matrix defined in our thesis to perform the information retrieval effectively. In the defined fuzzy term matrix, all relation degrees between terms are inferred from using the transitive closure algorithm to reflect all implicit links between terms into processing of the information retrieval. With this framework, the CLTR system proposed in our thesis enhances the retrieval effectiveness because it is able to emulate a human expert's decision making well in CLTR.

• Annual Conference on Human and Language Technology
• /
• 2023.10a
• /
• pp.289-294
• /
• 2023

대형 언어 모델(Large Language Model)은 방대한 데이터와 파라미터를 기반으로 문맥 이해에서 높은 성능을 달성하였지만, Human Alignment를 위한 문장 생성 제어 연구는 아직 활발한 도전 과제로 남아있다. 본 논문에서는 Instruction Tuning을 통한 문장 생성 제어 실험을 진행한다. 자연어 처리 도구를 사용하여 단일 혹은 다중 제약 조건을 포함하는 Instruction 데이터 셋을 자동으로 구축하고 한국어 언어 모델인 Polyglot-Ko 모델에 fine-tuning 하여 모델 생성이 제약 조건을 만족하는지 검증하였다. 실험 결과 4개의 제약 조건에 대해 평균 0.88의 accuracy를 보이며 효과적인 문장 생성 제어가 가능함을 확인하였다.

• Journal of Information Management
• /
• v.27 no.4
• /
• pp.35-56
• /
• 1996

SGML(Standard Generalized Markup Language) and its application to full-text database including a table, a figure and a picture are explained. A structure of SGML based full-text database Is defined by DTD(document type definition) written in SGML, and full-text itself is described with generalized markup depending on DTD. This article explains how to represent a document structure : a hierarchical structure like a chapter, a section, or a paragraph, or non-hierarchical(referencial) structure like a note, a table, a figure or a picture. Merits of SGML, electronic publishing, a retrieval system or hypertext and SGML tools are also described.