• Journal of the Korean Association of Geographic Information Studies
• /
• v.27 no.1
• /
• pp.115-127
• /
• 2024

This research aimed to construct models with various structures based on the Transformer module and to perform land cover classification, thereby examining the applicability of the Transformer module. For the classification of land cover, the Unet model, which has a CNN structure, was selected as the base model, and a total of four deep learning models were constructed by combining both the encoder and decoder parts with the Transformer module. During the training process of the deep learning models, the training was repeated 10 times under the same conditions to evaluate the generalization performance. The evaluation of the classification accuracy of the deep learning models showed that the Model D, which utilized the Transformer module in both the encoder and decoder structures, achieved the highest overall accuracy with an average of approximately 89.4% and a Kappa coefficient average of about 73.2%. In terms of training time, models based on CNN were the most efficient. however, the use of Transformer-based models resulted in an average improvement of 0.5% in classification accuracy based on the Kappa coefficient. It is considered necessary to refine the model by considering various variables such as adjusting hyperparameters and image patch sizes during the integration process with CNN models. A common issue identified in all models during the land cover classification process was the difficulty in detecting small-scale objects. To improve this misclassification phenomenon, it is deemed necessary to explore the use of high-resolution input data and integrate multidimensional data that includes terrain and texture information.

• ETRI Journal
• /
• v.46 no.1
• /
• pp.137-153
• /
• 2024

This study introduces a new approach in Korean morphological analysis combining dictionary-based techniques with Transformer-based deep learning models. The key innovation is the use of a BERT-based reranking system, significantly enhancing the accuracy of traditional morphological analysis. The method generates multiple suboptimal paths, then employs BERT models for reranking, leveraging their advanced language comprehension. Results show remarkable performance improvements, with the first-stage reranking achieving over 20% improvement in error reduction rate compared with existing models. The second stage, using another BERT variant, further increases this improvement to over 30%. This indicates a significant leap in accuracy, validating the effectiveness of merging dictionary-based analysis with contemporary deep learning. The study suggests future exploration in refined integrations of dictionary and deep learning methods as well as using probabilistic models for enhanced morphological analysis. This hybrid approach sets a new benchmark in the field and offers insights for similar challenges in language processing applications.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.05a
• /
• pp.624-626
• /
• 2023

Credit scoring is a technique used by financial institutions to assess the creditworthiness of potential borrowers. This involves evaluating a borrower's credit history to predict the likelihood of defaulting on a loan. This paper presents an ensemble of two Transformer based models within a framework for discriminating the default risk of loan applications in the field of credit scoring. The first model is FinBERT, a pretrained NLP model to analyze sentiment of financial text. The second model is FT-Transformer, a simple adaptation of the Transformer architecture for the tabular domain. Both models are trained on the same underlying data set, with the only difference being the representation of the data. This multi-modal approach allows us to leverage the unique capabilities of each model and potentially uncover insights that may not be apparent when using a single model alone. We compare our model with two famous ensemble-based models, Random Forest and Extreme Gradient Boosting.

• ETRI Journal
• /
• v.44 no.5
• /
• pp.794-804
• /
• 2022

Transformer models have had a great impact on natural language processing (NLP) in recent years by realizing outstanding and efficient contextualized language models. Recent studies have used transformer-based language models for various NLP tasks, including Persian named entity recognition (NER). However, in complex tasks, for example, NER, it is difficult to determine which contextualized embedding will produce the best representation for the tasks. Considering the lack of comparative studies to investigate the use of different contextualized pretrained models with sequence modeling classifiers, we conducted a comparative study about using different classifiers and embedding models. In this paper, we use different transformer-based language models tuned with different classifiers, and we evaluate these models on the Persian NER task. We perform a comparative analysis to assess the impact of text representation and text classification methods on Persian NER performance. We train and evaluate the models on three different Persian NER datasets, that is, MoNa, Peyma, and Arman. Experimental results demonstrate that XLM-R with a linear layer and conditional random field (CRF) layer exhibited the best performance. This model achieved phrase-based F-measures of 70.04, 86.37, and 79.25 and word-based F scores of 78, 84.02, and 89.73 on the MoNa, Peyma, and Arman datasets, respectively. These results represent state-of-the-art performance on the Persian NER task.

• Journal of Electrical Engineering and Technology
• /
• v.10 no.3
• /
• pp.1093-1101
• /
• 2015

Partial Discharge (PD) is a physical phenomenon, which causes defects and damages to the insulation. This phenomenon is regarded as the most important source of fault and defect in power transformers. Therefore, methods of high speed and precision are considered of special importance for the maintenance of transformers in localization of the origin of partial discharge. In this paper, the transformer winding is first modeled in a transient state by using RLC ladder network and multiconductor transmission line (MTL) models. The parameters of the two models were calculated by Ansoft Maxwell software, and the simulations were performed by Matlab software. Then, the PD pulses were applied to the models with different widths of pulses. With regard to the fact that the signals received after the application of PD had a variable frequency nature over time, and based on the wavelet transform and signal energy, a new method was presented for the localization of PD. Ultimately; the mentioned method was implemented on a 20 kV winding distribution transformer. Then, the performances of the models used in this paper, including RLC and MTL models, were compared in different frequency bands for the correct distinction of partial discharge location.

• Journal of Broadcast Engineering
• /
• v.27 no.7
• /
• pp.1011-1020
• /
• 2022

Recent high-performance panoptic segmentation models are based on transformer architectures. However, transformer-based panoptic segmentation methods are basically slower than convolution-based methods, since the attention mechanism in the transformer requires quadratic complexity w.r.t. image resolution. Also, sine and cosine computation for positional embedding in the transformer also yields a bottleneck for computation time. To address these problems, we adopt three modules to speed up the inference runtime of the transformer-based panoptic segmentation. First, we perform channel-level reduction using depth-wise separable convolution for inputs of the transformer decoder. Second, we replace sine and cosine-based positional encoding with convolution operations, called conv-embedding. We also apply a separable self-attention to the transformer encoder to lower quadratic complexity to linear one for numbers of image pixels. As result, the proposed model achieves 44% faster frame per second than baseline on ADE20K panoptic validation dataset, when we use all three modules.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.17 no.6
• /
• pp.319-327
• /
• 2022

Transformers are the most famous deep learning models that has achieved great success in natural language processing and also showed good performance on computer vision. In this survey, we categorized transformer-based models for computer vision, particularly object detection tasks and perform comprehensive comparative experiments to understand the characteristics of each model. Next, we evaluated the models subdivided into standard transformer, with key point attention, and adding attention with coordinates by performance comparison in terms of object detection accuracy and real-time performance. For performance comparison, we used two metrics: frame per second (FPS) and mean average precision (mAP). Finally, we confirmed the trends and relationships related to the detection and real-time performance of objects in several transformer models using various experiments.

• The Transactions of the Korean Institute of Power Electronics
• /
• v.27 no.6
• /
• pp.507-514
• /
• 2022

This study proposes an optimal design process for a high-frequency transformer that has a large leakage inductance for dual-active-bridge converters. Notably, conventional design processes have large errors in designing leakage transformers because mathematically modeling the leakage inductance of such transformers is difficult. In this work, the geometric parameters of a shell-type transformer are identified, and finite element analysis(FEA) simulation is performed to determine the magnetization inductance, leakage inductance, and copper loss of various shapes of shell-type transformers. Regression models for magnetization and leakage inductances and copper loss are established using the simulation results and the machine learning technique. In addition, to improve the regression models' performance, the regression models are tuned by adding featured parameters that consider the physical characteristics of the transformer. With the regression models, optimal high-frequency transformer designs and the Pareto front (in terms of volume and loss) are determined using NSGA-II. In the Pareto front, a desirable optimal design is selected and verified by FEA simulation and experimentation. The simulated and measured leakage inductances of the selected design match well, and this result shows the validity of the proposed design process.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.18 no.4
• /
• pp.693-700
• /
• 2023

Sentiment analysis, a branch of natural language processing that classifies and identifies subjective opinions and emotions in text documents as positive or negative, can be used for various promotions and services through customer preference analysis. To this end, recent research has been conducted utilizing various techniques in machine learning and deep learning. In this study, we propose an optimal language model by comparing the accuracy of sentiment analysis for movie, product, and game reviews using existing RNN-based models and recent Transformer-based language models. In our experiments, LMKorBERT and GPT3 showed relatively good accuracy among the models pre-trained on the Korean corpus.

• Journal of Korean Society of Industrial and Systems Engineering
• /
• v.46 no.3
• /
• pp.123-138
• /
• 2023

The Fourth Industrial Revolution and sensor technology have led to increased utilization of sensor data. In our modern society, data complexity is rising, and the extraction of valuable information has become crucial with the rapid changes in information technology (IT). Recurrent neural networks (RNN) and long short-term memory (LSTM) models have shown remarkable performance in natural language processing (NLP) and time series prediction. Consequently, there is a strong expectation that models excelling in NLP will also excel in time series prediction. However, current research on Transformer models for time series prediction remains limited. Traditional RNN and LSTM models have demonstrated superior performance compared to Transformers in big data analysis. Nevertheless, with continuous advancements in Transformer models, such as GPT-2 (Generative Pre-trained Transformer 2) and ProphetNet, they have gained attention in the field of time series prediction. This study aims to evaluate the classification performance and interval prediction of remaining useful life (RUL) using an advanced Transformer model. The performance of each model will be utilized to establish a health index (HI) for cutting blades, enabling real-time monitoring of machine health. The results are expected to provide valuable insights for machine monitoring, evaluation, and management, confirming the effectiveness of advanced Transformer models in time series analysis when applied in industrial settings.