KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Deep Learning Model Validation Method Based on Image Data Feature Coverage

영상 데이터 특징 커버리지 기반 딥러닝 모델 검증 기법

  • 임창남 (아주대학교 AI융합네트워크학과) ;
  • 박예슬 (아주대학교 AI융합네트워크학과) ;
  • 이정원 (아주대학교 전자공학과/AI융합네트워크학과)
  • Received : 2021.06.07
  • Accepted : 2021.07.04
  • Published : 2021.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Deep learning techniques have been proven to have high performance in image processing and are applied in various fields. The most widely used methods for validating a deep learning model include a holdout verification method, a k-fold cross verification method, and a bootstrap method. These legacy methods consider the balance of the ratio between classes in the process of dividing the data set, but do not consider the ratio of various features that exist within the same class. If these features are not considered, verification results may be biased toward some features. Therefore, we propose a deep learning model validation method based on data feature coverage for image classification by improving the legacy methods. The proposed technique proposes a data feature coverage that can be measured numerically how much the training data set for training and validation of the deep learning model and the evaluation data set reflects the features of the entire data set. In this method, the data set can be divided by ensuring coverage to include all features of the entire data set, and the evaluation result of the model can be analyzed in units of feature clusters. As a result, by providing feature cluster information for the evaluation result of the trained model, feature information of data that affects the trained model can be provided.

딥러닝 기법은 영상 처리 분야에서 높은 성능을 입증 받아 다양한 분야에서 적용되고 있다. 이러한 딥러닝 모델의 검증에 가장 널리 사용되는 방법으로는 홀드아웃 검증 방법, k-겹 교차 검증 방법, 부트스트랩 방법 등이 있다. 이러한 기존의 기법들은 데이터 셋을 분할하는 과정에서 클래스 간의 비율에 대한 균형을 고려하지만, 같은 클래스 내에서도 존재하는 다양한 특징들의 비율은 고려하지 않고 있다. 이러한 특징들을 고려하지 않을 경우, 일부 특징에 편향된 검증 결과를 얻게 될 수 있다. 따라서 본 논문에서는 기존 검증 방법들을 개선하여 영상 분류를 위한 데이터 특징 커버리지 기반의 딥러닝 모델 검증 기법을 제안한다. 제안하는 기법은 딥러닝 모델의 학습과 검증을 위한 훈련 데이터 셋과 평가 데이터 셋이 전체 데이터 셋의 특징을 얼마나 반영하고 있는지 수치로 측정할 수 있는 데이터 특징 커버리지를 제안한다. 이러한 방식은 전체 데이터 셋의 특징을 모두 포함하도록 커버리지를 보장하여 데이터 셋을 분할할 수 있고, 모델의 평가 결과를 생성한 특징 군집 단위로 분석할 수 있다. 검증결과, 훈련 데이터 셋의 데이터 특징 커버리지가 낮아질 경우, 모델이 특정 특징에 편향되게 학습하여 모델의 성능이 낮아지며, Fashion-MNIST의 경우 정확도가 8.9%까지 차이나는 것을 확인하였다.

Keywords

Acknowledgement

본 논문은 산업통상자원부 및 한국산업기술진흥원의 창의산업기술개발기반 구축사업의 일환으로 수행하였음(N0002312, 디지털 헬스케어 소프트웨어 시험평가센터 구축). 이 논문은 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(No. 2020R1A2C1007400).

References

  1. A. Esteva, et al, "Dermatologist-level classification of skin cancer with deep neural networks," Nature, Vol.542, No.7639, pp.115-118, 2017. https://doi.org/10.1038/nature21056
  2. F. Milletari, N. Navab, and S. A. Ahmadi, "V-net: Fully convolutional neural networks for volumetric medical image segmentation," 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016.
  3. A. Angelova, A. Krizhevsky, V. Vanhoucke, A. Ogale, and D. Ferguson, "Real-time pedestrian detection with deep network cascades," 2015.
  4. M. Bojarski, et al., "End to end learning for self-driving cars," arXiv preprint arXiv:1604.07316, 2016.
  5. A. Ferdowsi and W. Saad, "Deep learning for signal authentication and security in massive internet-of-things systems," IEEE Transactions on Communications, Vol.67, No.2, pp.1371-1387, 2018. https://doi.org/10.1109/tcomm.2018.2878025
  6. R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," Ijcai, Vol.14. No.2, pp.1137-1145, 1995.
  7. S. Mani, A. Sankaran, S. Tamilselvam, and A. Sethi, "Coverage testing of deep learning models using dataset characterization," arXiv preprint arXiv:1911.07309, 2019.
  8. T. Lindeberg, "Scale invariant feature transform," pp.10491, 2012.
  9. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol.1. IEEE, 2005.
  10. R. Lienhart and J. Maydt, "An extended set of haar-like features for rapid object detection," Proceedings. International Conference on Image Processing, Vol.1. IEEE, 2002.
  11. M. Flickner, et al, "Query by image and video content: The QBIC system," Computer, Vol.28, No.9, pp.23-32, 1995. https://doi.org/10.1109/2.410146
  12. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," arXiv preprint arXiv:1312.6229, 2013.
  13. A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014.
  14. L. Liu, C. Shen, and A. Van Den Hengel, "The treasure beneath convolutional layers: Cross-convolutional-layer pooling for image classification," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
  15. H. Xiao, K. Rasul, and R. Vollgraf, "Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms," arXiv preprint arXiv:1708.07747, 2017.
  16. A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," p.7, 2009.
  17. S. Yadav and S. Shukla, "Analysis of k-fold cross-validation over hold-out validation on colossal datasets for quality classification," 2016 IEEE 6th International Conference on Advanced Computing (IACC), IEEE, 2016.
  18. Y. Xu and R. Goodacre, "On splitting training and validation set: A comparative study of cross-validation, bootstrap and systematic sampling for estimating the generalization performance of supervised learning," Journal of Analysis and Testing, Vol.2, No.3, pp.249-262, 2018. https://doi.org/10.1007/s41664-018-0068-2
  19. M. Ozuysal, M. Calonder, V. Lepetit, and P. Fua, "Fast keypoint recognition using random ferns," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.32, No.3, pp.448-461, 2009. https://doi.org/10.1109/TPAMI.2009.23
  20. T. Ojala, M. Pietikainen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern Recognition, Vol.29, No.1, pp.51-59, 1996. https://doi.org/10.1016/0031-3203(95)00067-4
  21. B. Froba and A. Ernst, "Face detection with the modified census transform," Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings, IEEE, 2004.
  22. G. Chandrashekar and F. Sahin, "A survey on feature selection methods," Computers & Electrical Engineering, Vol.40, No.1, pp.16-28, 2014. https://doi.org/10.1016/j.compeleceng.2013.11.024
  23. D. ping Tian, "A review on image feature extraction and representation techniques," International Journal of Multimedia and Ubiquitous Engineering, Vol.8, No.4, pp.385-396, 2013.
  24. A. K. Jain and A. Vailaya, "Image retrieval using color and shape," Pattern Recognition, Vol.29, No.8, pp.1233-1244, 1996. https://doi.org/10.1016/0031-3203(95)00160-3
  25. G. Pass, and R. Zabih, "Histogram refinement for content-based image retrieval," Proceedings Third IEEE Workshop on Applications of Computer Vision, WACV'96. IEEE, 1996.
  26. J. Huang, S.R. Kumar, M. Mitra, W. Zhu, and R. Zabih, "Image indexing using color correlograms," Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 1997.
  27. L. Rokach and O. Maimon, "Clustering methods," Data Mining and Knowledge Discovery Handbook. Springer, Boston, MA, pp.321-352, 2005.
  28. S. Na, L. Xumin, and G. Yong, "Research on k-means clustering algorithm: An improved k-means clustering algorithm," 2010 Third International Symposium on intelligent Information Technology and Security Informatics, IEEE, 2010.
  29. R. Xu and D. Wunsch, "Survey of clustering algorithms," IEEE Transactions on Neural Networks, Vol.16, No.3, pp.645-678, 2005. https://doi.org/10.1109/TNN.2005.845141
  30. R. Bro and A. K. Smilde, "Principal component analysis," Analytical Methods, Vol.6, No.9, pp.2812-2831, 2014. https://doi.org/10.1039/c3ay41907j
  31. J. Xue, C. Lee, S. G.Wakeham, and R. A. Armstronga, "Using principal components analysis (PCA) with cluster analysis to study the organic geochemistry of sinking particles in the ocean," Organic Geochemistry, Vol.42, No.4, pp.356-367, 2011. https://doi.org/10.1016/j.orggeochem.2011.01.012
  32. T. M. Kodinariya and P. R. Makwana, "Review on determining number of Cluster in K-Means Clustering," International Journal, Vol.1, No.6, pp.90-95, 2013.