Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A deep learning-based vision enhancement method for UAV assisted visual inspection of concrete cracks

  • Qi, Yanzhi (Department of Disaster Mitigation for Structures, Tongji University) ;
  • Yuan, Cheng (Department of Disaster Mitigation for Structures, Tongji University) ;
  • Kong, Qingzhao (Department of Disaster Mitigation for Structures, Tongji University) ;
  • Xiong, Bing (State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University) ;
  • Li, Peizhen (Department of Disaster Mitigation for Structures, Tongji University)
  • Received : 2020.11.13
  • Accepted : 2021.03.17
  • Published : 2021.06.25
⟨ Previous Next ⟩

Abstract

Implementing unmanned aerial vehicles (UAVs) on concrete surface-crack inspection leads to a promising visual crack detection approach. One of the challenges for automated field visual cracking inspection is image degradation caused by the rain or fog and motion blur during data acquisition. The present study combines two deep neural networks to address the image degradation problem. By using the Variance of Laplacian algorithm for quantifying image clarity, the proposed deep neural networks can remarkably enhance the sharpness of the degraded images. After vision enhancement process, Mask Region Convolutional Neutral Network (Mask R-CNN) was developed to perform automated crack identification and segmentation. Results show a 8~13% enhancement in prediction accuracy compared to the degraded images, indicating that the proposed deep learning-based vision enhancement method can effectivey identify and segment concrete surface cracks from photos captured by UAVs.

Keywords

Acknowledgement

This research is supported by the National Key Research and Development Program of China (Grant No. 2018YFC0705602), Science and Technology Commission of Shanghai Municipality (STCSM) (Grant No. 19DZ1201200), and China National Science Foundation (Grant No. 51978507).

References

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
  2. Agarwal, S. and Singh, D. (2015), "An adaptive statistical approach for non-destructive underline crack detection of ceramic tiles using millimeter wave imaging radar for industrial application", IEEE Sensors J., 15(12), 7036-7044. https://doi.org/10.1109/JSEN.2015.2469157
  3. Ali-Dib, M., Menou, K., Jackson, A.P., Zhu, C. and Hammond, N. (2020), "Automated crater shape retrieval using weakly-supervised deep learning", Icarus, 345, 113749. https://doi.org/10.1016/j.icarus.2020.113749
  4. Alla, S. and Asadi, S.S. (2020), "Integrated methodology of structural health monitoring for civil structures", Mater. Today: Proceedings, 27(2), 1066-1072. https://doi.org/10.1016/j.matpr.2020.01.435
  5. Bang, S., Park, S., Kim, H., Yoon, Y. and Kim, H. (2018), "A deep residual network with transfer learning for pixel-level road crack detection", Proceedings of the International Symposium on Automation and Robotics in Construction, Berlin, Germany, July.
  6. Bansal, R., Raj, G. and Choudhury, T. (2016), "Blur image detection using Laplacian operator and Open-CV", Proceedings of 5th International Conference on System Modeling & Advancement in Research Trends, pp. 63-67, Moradabad, India, November.
  7. Cha, Y.J., Choi, W., Suh, G. and Mahmoudkhani, S. (2017a), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastruct. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  8. Cha, Y.-J., Choi, W. and Buyukozturk, O. (2017b), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastruct. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  9. Dorafshan, S., Thomas, R.J. and Maguire, M. (2018), "Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete", Constr. Build. Mater., 186, 1031-1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011
  10. Dung, C.V. and Anh, L.D. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  11. Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T. and Mizutani, K. (2017), "State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow's Intelligent Network Traffic Control Systems", IEEE Commun. Surveys Tutor., 19(4), 2432-2455. https://doi.org/10.1109/COMST.2017.2707140
  12. Fan, G., Li, J. and Hao, H. (2019), "Lost data recovery for structural health monitoring based on convolutional neural networks", Struct. Contorl Health Monitor., 26(10), e2433. https://doi.org/10.1002/stc.2433
  13. Fang, F., Li, L., Gu, Y., Zhu, H. and Lim, J.-H. (2020), "A novel hybrid approach for crack detection", Pattern Recogn., 107, 107474. https://doi.org/10.1016/j.patcog.2020.107474
  14. Farhodov, X., Kwon, O.H., Kang, K.W., Lee, S. and Kwon, K. (2019), "Faster RCNN detection based OpenCV CSRT tracker using drone data", Proceedings of International Conference on Information Science and Communications Technologies, Tashkent, Uzbekistan, February. https://doi.org/10.1109/ICISCT47635.2019.9012043
  15. Fujio, Y., Xu, C.-N., Sakata, Y., Ueno, N. and Terasaki, N. (2020), "Invisible crack visualization and depth analysis by mechanoluminescence film", J. Alloys Compounds, 832, 154900. https://doi.org/10.1016/j.jallcom.2020.154900
  16. Ganesh, P., Volle, K., Volle, K., Burks, T.F., Burks, T.F., Mehta, S.S. and Mehta, S.S. (2019), "Deep orange: Mask R-CNN based orange detection and segmentation", IFAC-PapersOnLine, 52(30), 70-75. https://doi.org/10.1016/j.ifacol.2019.12.499
  17. Gavilan, M., Balcones, D., Marcos, O., Llorca, D. F., Sotelo, M. A., Parra, I., Ocana, M., Aliseda, P., Yarza, P. and Amirola, A. (2011), "Adaptive road crack detection system by pavement classification", Sensors (Basel), 11(10), 9628-9657. https://doi.org/10.3390/s111009628
  18. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, OH, USA, June.
  19. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014), "Generative adversarial nets", Proceedings of the 28th Conference on Neural Information Processing Systems (NIPS), Montreal, Canada, December.
  20. Hanzaei, S.H., Afshar, A. and Barazandeh, F. (2017), "Automatic detection and classification of the ceramic tiles' surface defects", Pattern Recogn., 66, 174-189. https://doi.org/10.1016/j.patcog.2016.11.021
  21. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, December.
  22. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June.
  23. He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017), "Mask RCNN", Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, October.
  24. Huang, X., Liu, Z., Zhang, X., Kang, J. and Guo, Y.J.M. (2020), "Surface damage detection for steel wire ropes using deep learning and computer vision techniques", Measurement, 161, 107843. https://doi.org/10.1016/j.measurement.2020.107843
  25. Ioffe, S. and Szegedy, C. (2015), "Batch normalization: Accelerating deep network training by reducing internal covariate shift", Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France, July.
  26. Islam, M.M. and Kim, J.M. (2019), "Vision-based autonomous crack detection of concrete structures using a fully convolutional encoder-decoder network", Sensors (Basel), 19(19), 4251. https://doi.org/10.3390/s19194251
  27. Ji, A., Xue, X., Wang, Y., Luo, X. and Xue, W. (2020), "An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement", Automat. Constr., 114, 103176. https://doi.org/10.1016/j.autcon.2020.103176
  28. Jia, W., Tian, Y., Luo, R., Zhang, Z., Lian, J. and Zheng, Y. (2020), "Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot", Comput. Electron. Agric., 172, 105380. https://doi.org/10.1016/j.compag.2020.105380.
  29. Kapela, R., Sniataa, P., Turkot, A., Rybarczyk, A., Pozarycki, A., Rydzewski, P., Wyczaek, M. and Bloch, A. (2015), "Asphalt surfaced pavement cracks detection based on histograms of oriented gradients", Proceedings of the 22nd International Conference Mixed Design of Integrated Circuits & Systems (MIXDES), pp. 579-584. https://doi.org/10.1109/MIXDES.2015.7208590
  30. Koch, C., Georgieva, K., Kasireddy, V., Akinci, B. and Fieguth, P. (2015), "A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure", Adv. Eng. Informat., 29(2), 196-210. https://doi.org/10.1016/j.aei.2015.01.008
  31. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012), "Imagenet classification with deep convolutional neural networks", Adv. Neural Info. Process. Syst., 25, 1097-1105. https://doi.org/10.1145/3065386
  32. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. and Shi, W. (2017), "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, July.
  33. Li, Y., Li, H. and Wang, H. (2018), "Pixel-wise crack detection using deep local pattern predictor for robot application", Sensors, 18(9), 3042. https://doi.org/10.3390/s18093042
  34. Liao, K.-W. and Lee, Y.-T. (2016), "Detection of rust defects on steel bridge coatings via digital image recognition", Automat. Constr., 71(2), 294-306. https://doi.org/10.1016/j.autcon.2016.08.008
  35. Liu, H. and Zhang, Y. (2019), "Image-driven structural steel damage condition assessment method using deep learning algorithm", Measurement, 133, 168-181. https://doi.org/10.1016/j.measurement.2018.09.081
  36. Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005
  37. Liu, Y., Yeoh, J.K. and Chua, D.K. (2020), "Deep Learning-Based Enhancement of Motion Blurred UAV Concrete Crack Images", J. Comput. Civil Eng., 34(5), 04020028. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000907
  38. Luo, D., Yue, Y., Li, P., Ma, J., Zhang, L., Ibrahim, Z. and Ismail, Z. (2016), "Concrete beam crack detection using tapered polymer optical fiber sensors", Measurement, 88, 96-103. https://doi.org/10.1016/j.measurement.2016.03.028
  39. Mohamed, Y.S., Shehata, H.M., Abdellatif, M. and Awad, T.H. (2019), "Steel crack depth estimation based on 2D images using artificial neural networks", Alexandria Eng. J., 58(4), 1167-1174. https://doi.org/10.1016/j.aej.2019.10.001
  40. Nhat-Duc, H., Nguyen, Q.-L. and Tran, V.-D. (2018), "Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network", Automat. Constr., 94, 203-213. https://doi.org/10.1016/j.autcon.2018.07.008
  41. Nishikawa, T., Yoshida, J., Sugiyama, T. and Fujino, Y. (2012), "Concrete crack detection by multiple sequential image filtering", Comput.-Aided Civil Infrastruct. Eng., 27(1), 29-47. https://doi.org/10.1111/j.1467-8667.2011.00716.x
  42. Ohsugi, H., Tabuchi, H., Enno, H. and Ishitobi, N. (2017), "Accuracy of deep learning, a machine-learning technology, using ultra-wide-field fundus ophthalmoscopy for detecting rhegmatogenous retinal detachment", Sci. Rep., 7, 9425. https://doi.org/10.1038/s41598-017-09891-x
  43. Ozgenel, C.F. (2017), "Concrete crack images for classification", Mendeley Data, v1.
  44. Park, S.E., Eem, S.-H. and Jeon, H. (2020), "Concrete crack detection and quantification using deep learning and structured light", Constr. Build. Mater., 252, 119096. https://doi.org/10.1016/j.conbuildmat.2020.119096
  45. Qian, R., Tan, R.T., Yang, W., Su, J. and Liu, J. (2018), "Attentive Generative Adversarial Network for Raindrop Removal from A Single Image", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June.
  46. Quintana, M., Torres, J. and Menendez, J.M. (2016), "A simplified computer vision system for road surface inspection and maintenance", IEEE Transact. Intell. Transport. Syst., 17(3), 608-619. https://doi.org/10.1109/TITS.2015.2482222
  47. Reddy, A., Indragandhi, V., Ravi, L. and Subramaniyaswamy, V. (2019), "Detection of cracks and damage in wind turbine blades using artificial intelligence-based image analytics", Measurement, 147, 106823. https://doi.org/10.1016/j.measurement.2019.07.051
  48. Ren, S., He, K., Girshick, R. and Sun, J. (2016), "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Transact. Pattern Anal. Mach. Intell., Barcelona, Spain, December.
  49. Silva, W.R.L.D. and Lucena, D.S.D. (2018), "Concrete cracks detection based on deep learning image classification", Proceedings of The 18th International Conference on Experimental Mechanics, 2(8), 489. https://doi.org/10.3390/ICEM18-05387
  50. Simonyan, K. and Zisserman, A. (2014), "Very deep convolutional networks for large-scale image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, June.
  51. Spencer Jr., B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Eng., 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  52. Varadharajan, S., Jose, S., Sharma, K., Wander, L. and Mertz, C. (2014), "Vision for road inspection", Proceedings of IEEE Winter Conf. on Applications of Computer Vision (WACV), Steamboat Springs, CO, USA, March, pp. 115-122. https://doi.org/10.1109/WACV.2014.6836111
  53. Wang, S.Y., Zhang, P.Z., Zhou, S.Y., Wei, D.B., Ding, F. and Li, F.K. (2020), "A computer vision based machine learning approach for fatigue crack initiation sites recognition", Computat. Mater. Sci., 171, 109259. https://doi.org/10.1016/j.commatsci.2019.109259
  54. Wu, L., Mokhtari, S., Nazef, A., Nam, B. and Yun, H.-B. (2016), "Improvement of crack-detection accuracy using a novel crack defragmentation technique in image-based road assessment", J. Comput. Civil Eng., 30(1), 04014118. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000451
  55. Xu, B., Wang, W., Falzon, G., Kwan, P., Guo, L., Chen, G., Tait, A. and Schneider, D. (2020), "Automated cattle counting using Mask R-CNN in quadcopter vision system", Comput. Electron. Agric., 171, 105300. https://doi.org/10.1016/j.compag.2020.105300
  56. Yang, Q., Shi, W., Chen, J. and Lin, W. (2020), "Deep convolution neural network-based transfer learning method for civil infrastructure crack detection", Automat. Constr., 116, 103199. https://doi.org/10.1016/j.autcon.2020.103199
  57. Yousaf, M.H., Azhar, K., Murtaza, F. and Hussain, F. (2018), "Visual analysis of asphalt pavement for detection and localization of potholes", Adv. Eng. Info., 38, 527-537. https://doi.org/10.1016/j.aei.2018.09.002
  58. Yu, C., Fan, X., Hu, Z., Xia, X., Zhao, Y., Li, R. and Bai, Y. (2020), "Segmentation and measurement scheme for fish morphological features based on Mask R-CNN", Info. Process. Agriculture, 7(4), 523-534. https://doi.org/10.1016/j.inpa.2020.01.002
  59. Zhou, Z., Wu, Q.J., Wan, S., Sun, W. and Sun, X. (2020), "Integrating SIFT and CNN feature matching for partial-duplicate image detection", IEEE Transact. Emerg. Topics Computat. Intell., 204(5), 593-604 https://doi.org/10.1109/TETCI.2019.2909936