• Journal of the Korea Society of Computer and Information
• /
• v.28 no.4
• /
• pp.31-39
• /
• 2023

In this paper, we propose an Internal/External Knowledge Distillation (IEKD), which utilizes both external correlations between feature maps of heterogeneous models and internal correlations between feature maps of the same model for transferring knowledge from a teacher model to a student model. To achieve this, we transform feature maps into a sequence format and extract new feature maps suitable for knowledge distillation by considering internal and external correlations through a transformer. We can learn both internal and external correlations by distilling the extracted feature maps and improve the accuracy of the student model by utilizing the extracted feature maps with feature matching. To demonstrate the effectiveness of our proposed knowledge distillation method, we achieved 76.23% Top-1 image classification accuracy on the CIFAR-100 dataset with the "ResNet-32×4/VGG-8" teacher and student combination and outperformed the state-of-the-art KD methods.

• Journal of the Korea Society of Computer and Information
• /
• v.29 no.10
• /
• pp.35-45
• /
• 2024

In this paper, we propose Multi-Scale Feature Knowledge Distillation for 3D Object Detection (M3KD), which extracting knowledge from the teacher model, and transfer to the student model consider with multi-scale feature map. To achieve this, we minimize L2 loss between feature maps at each pyramid level of the student model with the correspond teacher model so student model can mimic the teacher model backbone information which improves the overall accuracy of the student model. We apply the class logits knowledge distillation used in the image classification task, by allowing student model mimic the classification logits of the teacher model, to guide the student model to improve the detection accuracy. In KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) dataset, our M3KD (Multi-Scale Feature Knowledge Distillation for 3D Object Detection) student model achieves 30% inference speed improvement compared to the teacher model. Additionally, our method achieved an average improvement of 1.08% in 3D mean Average Precision (mAP) across all classes and difficulty levels compared to the baseline student model. Furthermore, when integrated with the latest knowledge distillation methods such as PKD and SemCKD, our approach achieved an additional 0.42% and 0.52% improvement in 3D mAP, respectively, further enhancing performance.

• Journal of Digital Policy
• /
• v.2 no.3
• /
• pp.1-7
• /
• 2023

In this paper, I proposes a method for performing single image super resolution by separating texture-spatial domains and then classifying features based on detailed information. In CNN (Convolutional Neural Network) based super resolution, the complex procedures and generation of redundant feature information in feature estimation process for enhancing details can lead to quality degradation in super resolution. The proposed method reduced procedural complexity and minimizes generation of redundant feature information by splitting input image into two channels: texture and spatial. In texture channel, a feature refinement process with step-wise skip connections is applied for detail restoration, while in spatial channel, a method is introduced to preserve the structural features of the image. Experimental results using proposed method demonstrate improved performance in terms of PSNR and SSIM evaluations compared to existing super resolution methods, confirmed the enhancement in quality.

• Journal of Korea Multimedia Society
• /
• v.23 no.8
• /
• pp.1019-1029
• /
• 2020

In this paper, we propose a novel knowledge distillation algorithm to create an compressed deep ensemble network coupled with the combined use of local and global features of face images. In order to transfer the capability of high-level recognition performances of the ensemble deep networks to a single deep network, the probability for class prediction, which is the softmax output of the ensemble network, is used as soft target for training a single deep network. By applying the knowledge distillation algorithm, the local feature informations obtained by training the deep ensemble network using facial subregions of the face image as input are transmitted to a single deep network to create a so-called compressed ensemble DCNN. The experimental results demonstrate that our proposed compressed ensemble deep network can maintain the recognition performance of the complex ensemble deep networks and is superior to the recognition performance of a single deep network. In addition, our proposed method can significantly reduce the storage(memory) space and execution time, compared to the conventional ensemble deep networks developed for face recognition.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2020.11a
• /
• pp.226-227
• /
• 2020

근래의 초해상화 (super-resolution, SR) 연구는 네트워크를 깊고, 넓게 만들어 성능을 높이는데 주를 이뤘다. 그러나 동시에 높은 연산량과 메모리 소비량이 증가하는 문제가 발생하기 때문에 이를 실제로 하드웨어로 구현하기에는 어려운 문제가 존재한다. 그렇기에 우리는 네트워크 최적화를 통해 성능 감소를 최소화하면서 파라미터 수를 줄이는 네트워크 SqueezeSR을 설계하였다. 또한 지식 증류(Knowledge Distillation, KD)를 이용해 추가적인 파라미터 수 증가 없이 성능을 높일 수 있는 학습 방법을 제안한다. 또한 KD 시 teacher network의 성능이 보다 student network에 잘 전달되도록 feature map 간의 비교를 통해 학습 효율을 높일 수 있었다. 결과적으로 우리는 KD 기법을 통해 추가적인 파라미터 수 증가 없이 성능을 높여 다른 SR네트워크 대비 더 빠르고 성능 감소를 최소화한 네트워크를 제안한다.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.15 no.11
• /
• pp.4065-4083
• /
• 2021

Super-resolution can improve the clarity of low-resolution (LR) images, which can increase the accuracy of high-level compute vision tasks. Portable devices have low computing power and storage performance. Large-scale neural network super-resolution methods are not suitable for portable devices. In order to save the computational cost and the number of parameters, Lightweight image processing method can improve the processing speed of portable devices. Therefore, we propose the Enhanced Information Multiple Distillation Network (EIMDN) to adapt lower delay and cost. The EIMDN takes feedback mechanism as the framework and obtains low level features through high level features. Further, we replace the feature extraction convolution operation in Information Multiple Distillation Block (IMDB), with Ghost module, and propose the Enhanced Information Multiple Distillation Block (EIMDB) to reduce the amount of calculation and the number of parameters. Finally, coordinate attention (CA) is used at the end of IMDB and EIMDB to enhance the important information extraction from Spaces and channels. Experimental results show that our proposed can achieve convergence faster with fewer parameters and computation, compared with other lightweight super-resolution methods. Under the condition of higher peak signal-to-noise ratio (PSNR) and higher structural similarity (SSIM), the performance of network reconstruction image texture and target contour is significantly improved.

• Korean Chemical Engineering Research
• /
• v.58 no.4
• /
• pp.565-572
• /
• 2020

This study developed a software platform using machine learning of artificial intelligence to optimize the distillation column system. The distillation column is representative and core process in the petrochemical industry. Process stabilization is difficult due to various operating conditions and continuous process characteristics, and differences in process efficiency occur depending on operator skill. The process control based on the theoretical simulation was used to overcome this problem, but it has a limitation which it can't apply to complex processes and real-time systems. This study aims to develop an empirical simulation model based on machine learning and to suggest an optimal process operation method. The development of empirical simulations involves collecting big data from the actual process, feature extraction through data mining, and representative algorithm for the chemical process. Finally, the platform for the distillation column was developed with verification through a developed model and field tests. Through the developed platform, it is possible to predict the operating parameters and provided optimal operating conditions to achieve efficient process control. This study is the basic study applying the artificial intelligence machine learning technique for the chemical process. After application on a wide variety of processes and it can be utilized to the cornerstone of the smart factory of the industry 4.0.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.18 no.2
• /
• pp.51-58
• /
• 2023

This paper proposes a deep learning method for estimating the heart rate from facial videos. Our proposed method estimates remote photoplethysmography (rPPG) signals to predict the heart rate. Although there have been proposed several methods for estimating rPPG signals, most previous methods can not be utilized in low-power single board computers due to their computational complexity. To address this problem, we construct a lightweight student model and employ a knowledge distillation technique to reduce the performance degradation of a deeper network model. The teacher model consists of 795k parameters, whereas the student model only contains 24k parameters, and therefore, the inference time was reduced with the factor of 10. By distilling the knowledge of the intermediate feature maps of the teacher model, we improved the accuracy of the student model for estimating the heart rate. Experiments were conducted on the UBFC-rPPG dataset to demonstrate the effectiveness of the proposed method. Moreover, we collected our own dataset to verify the accuracy and processing time of the proposed method on a real-world dataset. Experimental results on a NVIDIA Jetson Nano board demonstrate that our proposed method can infer the heart rate in real time with the mean absolute error of 2.5183 bpm.