Performance Evaluation of Machine Learning and Deep Learning Algorithms in Crop Classification: Impact of Hyper-parameters and Training Sample Size

작물분류에서 기계학습 및 딥러닝 알고리즘의 분류 성능 평가: 하이퍼파라미터와 훈련자료 크기의 영향 분석

  • Kim, Yeseul (Department of Geoinformatic Engineering, Inha University) ;
  • Kwak, Geun-Ho (Department of Geoinformatic Engineering, Inha University) ;
  • Lee, Kyung-Do (National Institute of Agriculture Sciences, Rural Development Administration) ;
  • Na, Sang-Il (National Institute of Agriculture Sciences, Rural Development Administration) ;
  • Park, Chan-Won (National Institute of Agriculture Sciences, Rural Development Administration) ;
  • Park, No-Wook (Department of Geoinformatic Engineering, Inha University)
  • 김예슬 (인하대학교 공간정보공학과) ;
  • 곽근호 (인하대학교 공간정보공학과) ;
  • 이경도 (농촌진흥청 국립농업과학원) ;
  • 나상일 (농촌진흥청 국립농업과학원) ;
  • 박찬원 (농촌진흥청 국립농업과학원) ;
  • 박노욱 (인하대학교 공간정보공학과)
  • Received : 2018.09.28
  • Accepted : 2018.10.20
  • Published : 2018.10.31


The purpose of this study is to compare machine learning algorithm and deep learning algorithm in crop classification using multi-temporal remote sensing data. For this, impacts of machine learning and deep learning algorithms on (a) hyper-parameter and (2) training sample size were compared and analyzed for Haenam-gun, Korea and Illinois State, USA. In the comparison experiment, support vector machine (SVM) was applied as machine learning algorithm and convolutional neural network (CNN) was applied as deep learning algorithm. In particular, 2D-CNN considering 2-dimensional spatial information and 3D-CNN with extended time dimension from 2D-CNN were applied as CNN. As a result of the experiment, it was found that the hyper-parameter values of CNN, considering various hyper-parameter, defined in the two study areas were similar compared with SVM. Based on this result, although it takes much time to optimize the model in CNN, it is considered that it is possible to apply transfer learning that can extend optimized CNN model to other regions. Then, in the experiment results with various training sample size, the impact of that on CNN was larger than SVM. In particular, this impact was exaggerated in Illinois State with heterogeneous spatial patterns. In addition, the lowest classification performance of 3D-CNN was presented in Illinois State, which is considered to be due to over-fitting as complexity of the model. That is, the classification performance was relatively degraded due to heterogeneous patterns and noise effect of input data, although the training accuracy of 3D-CNN model was high. This result simply that a proper classification algorithms should be selected considering spatial characteristics of study areas. Also, a large amount of training samples is necessary to guarantee higher classification performance in CNN, particularly in 3D-CNN.


Supported by : 농촌진흥청


  1. Cai, Y., K. Guan, J. Peng, S. Wang, C. Seifert, B. Wardlow, and Z. Li, 2018. A high-performance and in-season classification system of field-level crop types using time-series Landsat data and a machine learning approach, Remote Sensing of Environment, 210: 35-47.
  2. Castro, J.B., R.Q. Feitosa, L.C.L. Rosa, P.A. Diaz, and I. Sanches, 2017. A comparative analysis of deep learning techniques for sub-tropical crop types recognition from multitemporal optical/SAR image sequences, Proc. of 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images, Niteroi, Oct. 17-20, pp. 382-389.
  3. Dang, L.M., S.I. Hassan, I. Suhyeon, A.K. Sangaiah, I. Mehmood, S. Rho, S. Seo, and H. Moon, 2018. UAV based wilt detection system via convolutional neural networks, Sustainable Computing: Informatics and Systems,
  4. Deng, L., Z. Mao, X. Li, Z. Hu, F. Duan, and Y. Yan, 2018. UAV-based multispectral remote sensing for precision agriculture: a comparison between different cameras, ISPRS Journal of Photogrammetry and Remote Sensing, 146: 124-136.
  5. Gao, Q., S. Lim, and X. Jia, 2018. Hyperspectral image classification using convolutional neural networks and multiple feature learning, Remote Sensing, 10(2): 299.
  6. Garcia-Garcia, A., J. Garcia-Rodriguez, S. Orts-Escolano, S. Oprea, F. Gomez-Donoso, and M. Cazorla, 2017. A study of the effect of noise and occlusion on the accuracy of convolutional neural networks applied to 3D object recognition, Computer Vision and image Understanding, 164: 124-134.
  7. Hall, O., S. Dahlin, H. Marstorp, M.F.A. Bustos, I. Oborn, and M. Jirstrom, 2018. Classification of maize in complex smallholder farming systems using UAV imagery, Remote Sensing, 2(3): 22.
  8. Hsu, C.-W., C.-C. Chang, and C.-J. Lin, 2003. A practical guide to support vector classification, Department of Computer Science, National Taiwan University, Taipei City, Taiwan.
  9. Ishida, T., J. Kurihara, F.A. Viray, S.B. Namuco, E.C. Paringit, G.J. Perez, Y. Takahashi, and J.J. Marciano Jr., 2018. A novel approach for vegetation classification using UAV-based hyperspectral imaging, Computers and Electronics in Agriculture, 144: 80-85.
  10. Ji, S., C. Zhang, A. Xu, Y. Shi, and Y. Duan, 2018. 3D convolutional neural networks for crop classification with multi-temporal remote sensing images, Remote Sensing, 10(1): 75.
  11. Ji, S., W. Xu, M. Yang, and K. Yu, 2013. 3D convolutional neural networks for human action recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1): 221-231.
  12. Kamilaris, A. and F.X. Prenafeta-Boldu, 2018. Deep learning in agriculture: a survey, Computers and Electronics in Agriculture, 147: 70-90.
  13. Karpathy, A., G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, 2014. Large-scale video classification with convolutional neural networks, Proc. of 2014 IEEE conference on Computer vision and Pattern Recognition, Columbus, OH, Jun. 23-28, pp. 1725-1732.
  14. Kim, Y., N.-W. Park, and K.-D. Lee, 2017. Self-learning based land-cover classification using sequential class patterns from past land-cover maps, Remote Sensing, 9(9): 921.
  15. Kussul, N., M. Lavreniuk, S. Skakun, and A. Shelestov, 2017. Deep learning classification of land cover and crop types using remote sensing data, IEEE Geoscience and Remote Sensing Letters, 14(5): 778-782.
  16. Kwak, G.-H., S. Park, H.Y. Yoo, and N.-W. Park, 2017. Updating land cover maps using object segmentation and past land cover information, Korean Journal of Remote Sensing, 33(6-2): 1089-1100 (in Korean with English abstract).
  17. LeCun, Y., Y. Bengio, and G. Hinton, 2015. Deep learning, Nature, 521(7553): 436.
  18. Lee, K.-D., Y.E. Lee, C.-W. Park, and S.-I. Na, 2016. A comparative study of image classification method to classify onion and garlic using unmanned aerial vehicle (UAV) imagery, Korean Society of Soil Science and Fertilizer, 49(6): 743-750 (in Korean with English abstract).
  19. Liu, T., A. Abd-Elrahman, J. Morton, and V.L. Wilhelm, 2018. Comparing fully convolutional networks, random forest, support vector machine, and patch-based deep convolutional neural networks for object-based wetland mapping using images from small unmanned aircraft system, GIScience & Remote Sensing, 55(2): 243-264.
  20. Maulik, U. and D. Chakraborty, 2017. Remote sensing image classification: a survey of support-vector-machine-based advanced techniques, IEEE Geoscience and Remote Sensing, 5(1): 33-52.
  21. Maxwell, A.E., T.A. Warner, and F. Fang, 2018. Implementation of machine-learning classification in remote sensing: an applied review, International Journal of Remote Sensing, 39(9): 2784-2817.
  22. Melgani, F. and L. Bruzzone, 2004. Classification of hyperspectral remote sensing images with support vector machines, IEEE Transactions on Geoscience and Remote Sensing, 42(8): 1778-1790.
  23. Montero, P. and J.A. Vilar, 2014. TSclust: an R package for time series clustering, Journal of Statistical Software, 62(1): 1-43.
  24. Onojeghuo, A.O., G.A. Blackburn, Q. Wang, P.M. Atkinson, D. Kindred, and Y. Miao, 2018. Mapping paddy rice fields by applying machine learning algorithms to multi-temporal Sentinel-1A and Landsat data, International Journal of Remote Sensing, 39(4): 1042-1067.
  25. Park, J.K. and J.H. Park, 2015. Crops classification using imagery of unmanned aerial vehicle (UAV), Journal of the Korean Society of Agricultural Engineers, 57(6): 91-97 (in Korean with English abstract).
  26. Prosekov, A.Y. and S.A. Ivanova, 2018. Food security: the challenge of the present, Geoforum, 91: 73-77.
  27. Sameen, M.I., B. Pradhan, and O.S. Aziz, 2018. Classification of very high resolution aerial photos using spectral-spatial convolutional neural networks, Journal of Sensors,
  28. Sharma, A., X. Liu, X. Yang, and D. Shi, 2017. A patch-based convolutional neural network for remote sensing image classification, Neural Networks, 95: 19-28.
  29. Sinha, T., B. Verma, and A. Haidar, 2017. Optimization of convolutional neural network parameters for image classification, Proc. of 2017 IEEE Symposium Series on Computational Intelligence, Honolulu, HI, Nov. 27-Dec. 1, pp. 1-7.
  30. Skakun, S., B. Franch, E. Vermote, J.-C. Roger, I. Becker-Reshef, C. Justice, and N. Kussul, 2017. Early season large-area winter crop mapping using MODIS NDVI data, growing degree days information and a Gaussian mixture model, Remote Sensing of Environment, 195: 244-258.
  31. Song, A. and Y. Kim, 2017. Deep learning-based hyperspectral image classification with application to environmental geographic information systems, Korean Journal of Remote Sensing, 33(6-2): 1061-1073 (in Korean with English abstract).
  32. Torbick, N., X. Huang, B. Ziniti, D. Johnson, J. Masek, and M. Reba, 2018. Fusion of moderate resolution earth observations for operational crop type mapping, Remote Sensing, 10(7): 1058.
  33. Wu, H. and S. Prasad, 2017. Convolutional recurrent neural networks for hyperspectral data classification, Remote Sensing, 9(3): 298.
  34. Xu, X., X. Ji, J. Jiang, X. Yao, Y. Tian, Y. Zhu, W. Cao, Q. Cao, H. Yang, Z. Shi, and T. Cheng, 2018. Evaluation of one-class support vector classification for mapping the paddy rice planting area in Jiangsu Province of China from Landsat 8 OLI imagery, Remote Sensing, 10(4): 546.
  35. Zhang, X., N. Chen, J. Li, Z. Chen, and D. Niyogi, 2017. Multi-sensor integrated framework and index for agricultural drought monitoring, Remote Sensing of Environment, 188: 141-163.
  36. Zhong, L., P. Gong, and G.S. Biging, 2014. Efficient corn and soybean mapping with temporal extendability: a multi-year experiment using Landsat imagery, Remote Sensing of Environment, 140: 1-13.
  37. Zhong, Z., J. Li, Z. Luo, and M. Chapman, 2018. Spectral-spatial residual network for hyperspectral image classification: a 3-D deep learning framework, IEEE Transactions on Geoscience and Remote Sensing, 56(2): 847-858.