Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Improved Method of License Plate Detection and Recognition using Synthetic Number Plate

인조 번호판을 이용한 자동차 번호인식 성능 향상 기법

  • Chang, Il-Sik (Dept. of Information Technology and Media Engineering, The graduate School of Nano IT Design Fusion, Seoul National University of Science and Technology) ;
  • Park, Gooman (Dept. of Electronic IT Media Engineering, Seoul National University of Science and Technology)
  • 장일식 (서울과학기술대학교 나노IT디자인융합대학원 정보통신미디어공학전공) ;
  • 박구만 (서울과학기술대학교 전자IT미디어공학과)
  • Received : 2021.05.28
  • Accepted : 2021.07.01
  • Published : 2021.07.30
Download PDF
⟨ Previous Next ⟩

Abstract

A lot of license plate data is required for car number recognition. License plate data needs to be balanced from past license plates to the latest license plates. However, it is difficult to obtain data from the actual past license plate to the latest ones. In order to solve this problem, a license plate recognition study through deep learning is being conducted by creating a synthetic license plates. Since the synthetic data have differences from real data, and various data augmentation techniques are used to solve these problems. Existing data augmentation simply used methods such as brightness, rotation, affine transformation, blur, and noise. In this paper, we apply a style transformation method that transforms synthetic data into real-world data styles with data augmentation methods. In addition, real license plate data are noisy when it is captured from a distance and under the dark environment. If we simply recognize characters with input data, chances of misrecognition are high. To improve character recognition, in this paper, we applied the DeblurGANv2 method as a quality improvement method for character recognition, increasing the accuracy of license plate recognition. The method of deep learning for license plate detection and license plate number recognition used YOLO-V5. To determine the performance of the synthetic license plate data, we construct a test set by collecting our own secured license plates. License plate detection without style conversion recorded 0.614 mAP. As a result of applying the style transformation, we confirm that the license plate detection performance was improved by recording 0.679mAP. In addition, the successul detection rate without image enhancement was 0.872, and the detection rate was 0.915 after image enhancement, confirming that the performance improved.

자동차 번호인식을 위해선 수많은 번호판 데이터가 필요하다. 번호판 데이터는 과거의 번호판부터 최신의 번호판까지 균형 있는 데이터의 확보가 필요하다. 하지만 실제 과거의 번호판부터 최신의 번호판의 데이터를 획득하는데 어려움이 있다. 이러한 문제를 해결하기 위해 인조 번호판을 이용하여 자동차 번호판을 생성하여 딥러닝을 통한 번호판 인식 연구가 진행되고 있다. 하지만 인조 데이터는 실제 데이터와 차이가 존재하며, 이러한 문제를 해결하기 위해 다양한 데이터 증강 기법을 사용한다. 기존 데이터 증강 방식은 단순히 밝기, 회전, 어파인 변환, 블러, 노이즈등의 방법을 사용하였다. 본 논문에서는 데이터 증강 방법으로 인조데이터를 실제 데이터 스타일로 변환하는 스타일 변환 방법을 적용한다. 또한 실제 번호판 데이터는 원거리가 많고 어두운 경우 잡음이 많이 존재한다. 단순히 입력데이터를 가지고 문자를 인식할 경우 오인식의 가능성이 높다. 이러한 경우 문자인식 향상을 위해 본 논문에서는 문자인식을 위하여 화질개선 방법으로 DeblurGANv2 방법을 적용하여 번호판 인식 정확도를 높였다. 번호판 검출 및 번호판 번호인식을 위한 딥러닝의 방식은 YOLO-V5를 사용하였다. 인조 번호판 데이터 성능을 판단하기 위해 자체적으로 확보한 자동차 번호판을 수집하여 테스트 셋을 구성하였다. 스타일 변환을 적용하지 않은 번호판 검출이 0.614mAP를 기록하였다. 스타일 변환을 적용한 결과 번호판 검출 성능이 0.679mAP 기록하여 성능이 향상되었음을 확인하였다. 또한 번호판 문자인식에는 화질 개선을 하지 않은 검출 성공률은 0.872를 기록하였으며, 화질 개선 후 검출 성능이 0.915를 기록하여 성능 향상이 되었음을 확인 하였다.

Keywords

Acknowledgement

This research was supported by R&BD Program through the INNOPOLIS funded by Ministry of Science and ICT (2020-IT-RD-0232).

References

  1. J. Van Hulse, T. M. Khoshgoftaar, A. Napolitano, "Experimental perspectives on learning from imbalanced data," in Proceedings of the ACM International Conference on Machine Learning, New York, pp.935-942, 2007.
  2. Jae-Hyeon Lee, Sung-Man Cho, Seung-Ju Lee, Cheong-Hwa Kim, Goo-Man Park. "License Plate Recognition System Using Synthetic Data," pp.107-115, 2020, doi:10.5573/ieie.2020.57.1.107
  3. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, et al. "Domain-Adversarial Training of Neural Networks," Journal of Machine Learning Research vol. 17, pp.1-35, 2016
  4. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. "Generative adversarial nets," Adv Neural Inf Process Syst. 2014.
  5. Connor Shorten, Taghi M, Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning," Journal of Big Data 2019.
  6. Terrance Devries, Graham W, Taylor, "Improved regularization of convolutional neural networks with Cutout," arXiv preprint arXiv:1708.04552, 2017.
  7. Hongyu Guo, Yongyi Mao, and Richong Zhang, "Mixup as locally linear out-of-manifold regularization," In AAAI, 2019.
  8. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," ICCV, 2019.
  9. Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan. "AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty," ICLR, 2020.
  10. Orest Kupyn, Tetiana Martyniuk, Junru Wu, Zhangyang Wang, "DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better," ICCV, 2019.
  11. Glenn Jocher.https://github.com/ultralytics/yolov5
  12. Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," arXiv: 1311.2524v5, Oct 2014.
  13. Ross Girshick, "Fast R-CNN," arXiv:1504.08083v2, Sep 2015.
  14. Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," arXiv: 1506.01497v3, Jan 2016.
  15. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: Single Shot MultiBox Detector," arXiv:1512.02325v5, Dec 2016.
  16. Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," arXiv:1506.02640v5, May 2016.
  17. Joseph Redmon, XNOR.ai, "YOLO9000: Better, Faster, Stronger," CVPR, 2017.
  18. Joseph Redmon, Ali Farhadi, "YOLOv3: An Incremental Improvement. arXiv:1804.02767, Apr 2018.
  19. Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection," arXiv:2004.10934, Apr 2020.
  20. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, "Image Style Transfer Using Convolutional Neural Networks," CVPR. 2016.
  21. Xun Huang Serge Belongie, "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization," ICCV. 2017.
  22. Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks," ICCV. 2017.
  23. Lee, Yu-Jin, Kim, Sang-Joon, Park, Gyeong-Moo, Park, GooMan, "Comparison of number plate recognition performance of Synthetic number plate generator using 2D and 3D rotation," The Korean Society Of Broad Engineers, pp.141-144, 2020.
  24. Ujjwal Saxena. https://github.com/UjjwalSaxena/Automold-Road-Augmentation-Library
  25. Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi, "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network," CVPR. 2017.
  26. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, "Enhanced Deep Residual Networks for Single Image Super-Resolution," CVPR. 2017.
  27. Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, Chen Change Loy, "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks," ECCV. 2018.
  28. Orest Kupyn, Volodymyr, et al. "DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks," CVPR. 2018.
  29. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, "Inception-v4, inception-resnet and the impact of residual connections on learning," In: Thirty-First AAAI Conference on Artificial Intelligence. 2017.
  30. Mark Sandler, Andrew Howard, Menglong Zhu, et al. "MobileNetV2: Inverted Residuals and Linear Bottlenecks," CVPR. 2018.