• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.24 no.1
• /
• pp.1-8
• /
• 2024

On-device AI technology, which can operate AI models at the edge devices to support real-time processing and privacy enhancement, is attracting attention. As intelligent IoT is applied to various industries, services utilizing the on-device AI technology are increasing significantly. However, general deep learning models require a lot of computational resources for inference and learning. Therefore, various lightweighting methods such as quantization and pruning have been suggested to operate deep learning models in embedded edge devices. Among the lightweighting methods, we analyze how to lightweight and apply deep learning models to edge computing devices, focusing on pruning technology in this paper. In particular, we utilize dynamic and static pruning techniques to evaluate the inference speed, accuracy, and memory usage of a lightweight AI vision model. The content analyzed in this paper can be used for intelligent video control systems or video security systems in autonomous vehicles, where real-time processing are highly required. In addition, it is expected that the content can be used more effectively in various IoT services and industries.

• Journal of the Institute of Convergence Signal Processing
• /
• v.24 no.1
• /
• pp.21-26
• /
• 2023

Knowledge distillation (KD) is a model lightening technology that transfers the knowledge of deep models to light models. Most KD methods have been developed for classification models, and there have been few KD studies in the field of super-resolution (SR). In this paper, various KD methods are applied to an SR model and their performance is compared. Specifically, we modified the loss function to apply each KD method to the SR model and conducted an experiment to learn a student model that was about 27 times lighter than the teacher model and to double the image resolution. Through the experiment, it was confirmed that some KD methods were not valid when applied to SR models, and that the performance was the highest when the relational KD and the traditional KD methods were combined.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.21 no.1
• /
• pp.93-98
• /
• 2021

Due to the development of deep learning and AI, the scale of the model has grown, and it has been integrated into other fields to blend into our lives. However, in environments with limited resources such as embedded devices, it is exist difficult to apply the model and problems such as power shortages. To solve this, lightweight methods such as clouding or offloading technologies, reducing the number of parameters in the model, or optimising calculations are proposed. In this paper, quantization of learned models is applied to ONNX models used in various framework interchange formats, neural network structure and inference performance are compared with existing models, and various module methods for quantization are analyzed. Experiments show that the size of weight parameter is compressed and the inference time is more optimized than before compared to the original model.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.05a
• /
• pp.530-531
• /
• 2023

딥러닝에서 지식 증류 기법은 큰 모델의 지식을 작은 모델로 전달하여 작은 모델의 성능을 개선하는 방식이다. 지식 증류 기법은 모델 경량화, 학습 속도 향상, 학습 정확도 향상 등에 활용될 수 있는데, 교사 모델이라 불리는 큰 모델은 일반적으로 학습된 딥러닝 모델을 사용한다. 본 연구에서는 학습된 딥러닝 모델 대신에 수치 기반 시뮬레이션 모델을 사용함으로써 어떠한 효과가 있는지 검증하였으며, 수치 모델을 활용한 기상 예측 모델에서의 지식 증류는 기존 단독 딥러닝 모델 학습 대비 더 작은 학습 횟수(epoch)에서도 동일한 에러 수준(RMSE)까지 도달하여, 학습 속도 측면에서 이득이 있음을 확인하였다.

• Journal of the Korea Society of Computer and Information
• /
• v.27 no.10
• /
• pp.29-41
• /
• 2022

Recently, with the development of deep learning technology, a variety of huge models with excellent performance have been devised by pre-training massive amounts of text data. However, in order for such a model to be applied to real-life services, the inference speed must be fast and the amount of computation must be low, so the technology for model compression is attracting attention. Knowledge distillation, a representative model compression, is attracting attention as it can be used in a variety of ways as a method of transferring the knowledge already learned by the teacher model to a relatively small-sized student model. However, knowledge distillation has a limitation in that it is difficult to solve problems with low similarity to previously learned data because only knowledge necessary for solving a given problem is learned in a teacher model and knowledge distillation to a student model is performed from the same point of view. Therefore, we propose a heterogeneous knowledge distillation method in which the teacher model learns a higher-level concept rather than the knowledge required for the task that the student model needs to solve, and the teacher model distills this knowledge to the student model. In addition, through classification experiments on about 18,000 documents, we confirmed that the heterogeneous knowledge distillation method showed superior performance in all aspects of learning efficiency and accuracy compared to the traditional knowledge distillation.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2018.10a
• /
• pp.705-707
• /
• 2018

Rotary Inverted Pendulum 은 제어분야에서 비선형 제어 시스템을 설명하기 위해 자주 사용되어왔다. 본 논문은 강화학습 에이전트의 환경으로써 Rotary Inverted Pendulum 을 도입하였다. 이를 통해서 강화학습이 실제 세계에서의 복합적인 문제를 해결할 수 있음을 보인다. 강화학습 에이전트의 가상 환경과 실제 환경을 맵핑시키기 위해서 Ethernet 연결 위에 MQTT 프로토콜을 사용하였으며 이를 통해서 경량화된 IoT 분야에서의 강화학습의 활용도를 조명한다.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• fall
• /
• pp.80-81
• /
• 2021

딥 러닝 모델 사용에 있어서, 일반적인 사용자가 이용할 수 있는 하드웨어 리소스는 제한적이기 때문에 기존 모델을 경량화 할 수 있는 프루닝 방법을 통해 제한적인 리소스를 효과적으로 활용할 수 있도록 한다. 그 방법으로, 여러 딥 러닝 모델들 중 비교적 파라미터 수가 많은 것으로 알려진 GAN 아키텍처에 네트워크 프루닝을 적용함으로써 비교적 무거운 모델을 적은 파라미터를 통해 학습할 수 있는 방법을 제시한다. 또한, 본 논문을 통해 기존의 SRGAN 논문에서 가장 효과적인 결과로 제시했던 16 개의 residual block 의 개수를 실제로 줄여 봄으로써 기존 논문에서 제시했던 결과와의 차이에 대해 서술한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2022.11a
• /
• pp.435-437
• /
• 2022

보행 보조 로봇의 효과적인 보조를 위해서는 사용자의 보행 유형을 인식하는 것이 중요하다. 본 논문에서는 end-to-end 분류 네트워크 기반 보행 환경 인식 방법을 사용하여 사용자의 보행 유형을 강인하게 추정한다. 실외 보행 환경을 오르막길, 평지, 내리막길 3 가지로 분류하는 딥러닝 모델을 학습시켰으며, 엣지 디바이스에서 이를 사용하기 위해 네트워크 경량화를 진행하였다. 경량화 후 추론 속도는 약 47FPS 수준으로 실시간으로 보행 보조 로봇에 적용 가능한 것을 검증했으며, 정확도 측면에서도 97% 이상의 성능을 얻을 수 있었다.

• Journal of the Institute of Convergence Signal Processing
• /
• v.22 no.3
• /
• pp.99-103
• /
• 2021

Recently, convolutional neural networks (CNNs) have been widely utilized due to their outstanding performance in various computer vision fields. However, due to their computational-intensive and high memory requirements, it is difficult to deploy CNNs on hardware platforms that have limited resources, such as mobile devices and IoT devices. To address these limitations, a neural network compression research is underway to reduce the size of neural networks while maintaining their performance. This paper proposes a CNN compression technique that dynamically adjusts the thresholds of pruning, one of the neural network compression techniques. Unlike the conventional pruning that experimentally or heuristically sets the thresholds that determine the weights to be pruned, the proposed technique can dynamically find the optimal thresholds that prevent accuracy degradation and output the light-weight neural network in less time. To validate the performance of the proposed technique, the LeNet was trained using the MNIST dataset and the light-weight LeNet could be automatically obtained 1.3 to 3 times faster without loss of accuracy.

• Annual Conference on Human and Language Technology
• /
• 2022.10a
• /
• pp.343-347
• /
• 2022

최근 자연어처리 분야에서는 BERT, RoBERTa, 그리고 BART와 같은 사전 학습된 언어 모델 (Pre-trained Language Models, PLM) 기반 미세 조정 학습을 통하여 여러 하위 과업에서 좋은 성능을 거두고 있다. 이는 사전 학습된 언어 모델 및 데이터 집합의 크기, 그리고 모델 구성의 중요성을 보여주며 대규모 사전 학습된 언어 모델이 각광받는 계기가 되었다. 하지만, 거대한 모델의 크기로 인하여 실제 산업에서 쉽게 쓰이기 힘들다는 단점이 명백히 존재함에 따라 최근 매개변수 효율적인 미세 조정 및 Few-Shot 학습 연구가 많은 주목을 받고 있다. 본 논문은 Prompt tuning, Prefix tuning와 프롬프트 기반 미세 조정 (Prompt-based fine-tuning)을 결합한 Few-Shot 학습 연구를 제안한다. 제안한 방법은 미세 조정 ←→ 사전 학습 간의 지식 격차를 줄일 뿐만 아니라 기존의 일반적인 미세 조정 기반 Few-Shot 학습 성능보다 크게 향상됨을 보인다.