IEMEK Journal of Embedded Systems and Applications (대한임베디드공학회논문지)

Institute of Embedded Engineering of Korea (대한임베디드공학회)
권호 표지
DOI QR코드

DOI QR Code

Multimodal Supervised Contrastive Learning for Crop Disease Diagnosis

멀티 모달 지도 대조 학습을 이용한 농작물 병해 진단 예측 방법

  • Received : 2023.09.20
  • Accepted : 2023.11.07
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

With the wide spread of smart farms and the advancements in IoT technology, it is easy to obtain additional data in addition to crop images. Consequently, deep learning-based crop disease diagnosis research utilizing multimodal data has become important. This study proposes a crop disease diagnosis method using multimodal supervised contrastive learning by expanding upon the multimodal self-supervised learning. RandAugment method was used to augment crop image and time series of environment data. These augmented data passed through encoder and projection head for each modality, yielding low-dimensional features. Subsequently, the proposed multimodal supervised contrastive loss helped features from the same class get closer while pushing apart those from different classes. Following this, the pretrained model was fine-tuned for crop disease diagnosis. The visualization of t-SNE result and comparative assessments of crop disease diagnosis performance substantiate that the proposed method has superior performance than multimodal self-supervised learning.

Keywords

Acknowledgement

이 성과는 정부 (과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임 (No. NRF-2021R1G1A1093546).

References

  1. K. P. Ferentinos, "Deep Learning Models for Plant Disease Detection and Diagnosis," Computers and Electronics in Agriculture, Vol. 145, pp. 311-318, 2018. https://doi.org/10.1016/j.compag.2018.01.009
  2. J. A. Pandian, V. D. Kumar, O. Geman, M. Hnatiuc, M. Arif, K. Kanchanadevi, "Plant Disease Detection Using Deep Convolutional Neural Network," Applied Sciences Vol. 12, No. 14, pp. 6982, 2022.
  3. H. Yu, J. Liu, C. Chen, A. A. Heidari, Q. Zhang, H. Chen, M. Mafarja, H. Turabieh, "Corn Leaf Diseases Diagnosis Based on K-means Clustering and Deep Learning," IEEE Access, Vol. 9, pp. 143824-143835, 2021. https://doi.org/10.1109/ACCESS.2021.3120379
  4. H. S. Yoon, S. B. Jeong, "Performance Comparison of Base CNN Models in Transfer Learning for Crop Diseases Classification," Journal of the Society of Korea Industrial and Systems Engineering, Vol. 44, No. 3, pp. 33-38, 2021. https://doi.org/10.11627/jkise.2021.44.3.033
  5. J. Li, Y. Qiao, S. Liu, J. Zhang, Z. Yang, M. Wang, "An Improved YOLOv5-based Vegetable Disease Detection Method," Computers and Electronics in Agriculture, Vol. 202, 107345, 2022.
  6. L. Alves, R. R. Silva, J. Bernardino, "System to Predict Diseases in Vineyards and Olive Groves using Data Mining and Geolocation," ICSOFT. 2018.
  7. S. Sannakki, V. S. Rajpurohit, F. Sumira, H. Venkatesh, "A Neural Network Approach for Disease Forecasting in Grapes Using Weather Parameters," 2013 Fourth International Conference on Computing, Communications and Networking Technologies (ICCCNT). IEEE, pp. 1-5, 2013.
  8. J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, F. F. Li, "Imagenet: A Large-scale Hierarchical Image Database," 2009 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, pp. 248-255, 2009.
  9. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y. Ng, "Multimodal Deep Learning," Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 689-696, 2011.
  10. J. Venugopalan, L. Tong, H. R. Hassanzadeh, M. D. Wang, "Multimodal Deep Learning Models for Early Detection of Alzheimer's Disease Stage," Scientific reports, Vol. 11, No. 1, pp. 3254, 2021.
  11. W. Liu, W. L. Zheng, B. L. Lu, "Emotion Recognition Using Multimodal Deep Learning," Neural Information Processing: 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16-21, 2016, Proceedings, Part II 23. Springer International Publishing, pp. 521-529, 2016.
  12. T. G. Kim, B. J. Kang, M. Rho, S. Sezer, E. G. Im, "A Multimodal Deep Learning Method for Android Malware Detection Using Various Features," IEEE Transactions on Information Forensics and Security, Vol. 14, No. 3, pp. 773-788, 2018. https://doi.org/10.1109/TIFS.2018.2866319
  13. T. Chen, S. Kornblith, M. Norouzi, G. Hinton, "A Simple Framework for Contrastive Learning of Visual Representations," International Conference on Machine Learning. PMLR, pp. 1597-1607, 2020.
  14. J. Zhang, J. Zou, Z. Su, J. Tang, Y. Kang, H. Xu, Z. Liu, S. Fan, "A Class-aware Supervised Contrastive Learning Framework for Imbalanced Fault Diagnosis," Knowledge-Based Systems, Vol. 252, 109437, 2022.
  15. A. Taleb, M. Kirchler, R. Monti, C. Lippert, "Contig: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20908-20921, 2022.
  16. P. Hager, M. J. Menten, D. Rueckert, "Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23924-23935, 2023.
  17. P. Khosla, P. Teterwak, C. Wang, "Supervised Contrastive Learning," Advances in Neural Information Processing Systems 33, pp. 18661-18673, 2020.
  18. DACON AI Challenge Website. accessed: 2023-05-12. https://dacon.io/competitions/official/235870/overview/description
  19. E. D. Cubuk, B. Zoph, J. Shlens, Q. V. Le, "Randaugment: Practical Automated Data Augmentation with a Reduced Search Space," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
  20. K. He, X. Zhang, S. Ren, J. Sun, "Deep Residual Learning for Image Recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
  21. B. K. Iwana, S. Uchida, "An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks," Plos one, Vol. 16, No. 7, e0254841, 2021.