Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

UAV-based bridge crack discovery via deep learning and tensor voting

  • Xiong Peng (Hunan Provincial Key Laboratory of Structures for Wind Resistance and Vibration Control, Hunan University of Science and Technology) ;
  • Bingxu Duan (Hunan Provincial Key Laboratory of Structures for Wind Resistance and Vibration Control, Hunan University of Science and Technology) ;
  • Kun Zhou (School of Civil Engineering, Hunan University of Science and Technology) ;
  • Xingu Zhong (Hunan Provincial Key Laboratory of Structures for Wind Resistance and Vibration Control, Hunan University of Science and Technology) ;
  • Qianxi Li (School of Civil Engineering, Hunan University of Science and Technology) ;
  • Chao Zhao (School of Civil Engineering, Hunan University of Science and Technology)
  • Received : 2022.11.22
  • Accepted : 2024.01.04
  • Published : 2024.02.25
⟨ Previous Next ⟩

Abstract

In order to realize tiny bridge crack discovery by UAV-based machine vision, a novel method combining deep learning and tensor voting is proposed. Firstly, the grid images of crack are detected and descripted based on SE-ResNet50 to generate feature points. Then, the probability significance map of crack image is calculated by tensor voting with feature points, which can define the direction and region of crack. Further, the crack detection anchor box is formed by non-maximum suppression from the probability significance map, which can improve the robustness of tiny crack detection. Finally, a case study is carried out to demonstrate the effectiveness of the proposed method in the Xiangjiang-River bridge inspection. Compared with the original tensor voting algorithm, the proposed method has higher accuracy in the situation of only 1-2 pixels width crack and the existence of edge blur, crack discontinuity, which is suitable for UAV-based bridge crack discovery.

Keywords

Acknowledgement

This study is supported by Chunhui Project Foundation of the Education Department of China (HZKY20220354) and Scientific Research Foundation of Hunan Provincial Education Department (23B0451), to which the authors are grateful.

References

  1. AASHTO (1970), Manual for Maintenance Inspection of Bridges, Washington, DC, USA.
  2. Ali, R., Kang, D., Suh, G. and Cha, Y.-J. (2021), "Real-time multiple damage mapping using autonomous UAV and deep faster region-based neural networks for GPS-denied structures", Automat. Constr., 130, 103831. https://doi.org/10.1016/j.autcon.2021.103831
  3. Bae, H., Jang, K. and An, Y.-K. (2021), "Deep super resolution crack network (SrcNet) for improving computer vision-based automated crack detectability in in situ bridges", Struct. Health Monitor., 20(4), 1428-1442. https://doi.org/10.1177/1475921720917227
  4. Bhowmick, S., Nagarajaiah, S. and Veeraraghavan, A. (2020), "Vision and deep learning-based algorithms to detect and quantify cracks on concrete surfaces from UAV videos", Sensors, 20(21), 6299. https://doi.org/10.3390/s20216299
  5. Bochkovskiy, A., Wang, C.-Y. and Liao, H.-Y.M. (2020), "Yolov4: Optimal speed and accuracy of object detection", arXiv preprint arXiv:2004.10934.
  6. Bolourian, N. and Hammad, A. (2020), "LiDAR-equipped UAV path planning considering potential locations of defects for bridge inspection", Automat. Constr., 117, 103250. https://doi.org/10.1016/j.autcon.2020.103250
  7. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  8. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2017a), "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Mach. Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184
  9. Chen, L.-C., Papandreou, G., Schroff, F. and Adam, H. (2017b), "Rethinking atrous convolution for semantic image segmentation", arXiv preprint arXiv:1706.05587.
  10. Chen, H., Zhao, H., Han, D., Liu, W., Chen, P. and Liu, K. (2020), "Structure-aware-based crack defect detection for multicrystalline solar cells", Measurement, 151, 107170. https://doi.org/10.1016/j.measurement.2019.107170
  11. Dai, J., Li, Y., He, K. and Sun, J. (2016), "R-fcn: Object detection via region-based fully convolutional networks", Adv. Neural Info. Process. Syst., 29.
  12. Deng, L., Chu, H.-H., Shi, P., Wang, W. and Kong, X. (2020), "Region-based CNN method with deformable modules for visually classifying concrete cracks", Appl. Sci., 10(7), 2528. https://doi.org/10.3390/app10072528
  13. Deng, J., Lu, Y. and Lee, V.C.-S. (2021), "Imaging-based crack detection on concrete surfaces using You Only Look Once network", Struct. Health Monitor., 20(2), 484-499. https://doi.org/10.1177/1475921720938486
  14. Dong, C.-Z. and Catbas, F.N. (2021), "A review of computer vision-based structural health monitoring at local and global levels", Struct. Health Monitor., 20(2), 692-743. https://doi.org/10.1177/1475921720935585
  15. Dorafshan, S. and Maguire, M. (2018), "Bridge inspection: human performance, unmanned aerial systems and automation", J. Civil Struct. Health Monitor., 8(3), 443-476. https://doi.org/10.1007/s13349-018-0285-4
  16. Dorafshan, S., Thomas, R.J. and Maguire, M. (2018), "Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete", Constr. Build. Mater., 186, 1031-1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011
  17. Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A. and Berg, A.C. (2017), "Dssd: Deconvolutional single shot detector", arXiv preprint arXiv:1701.06659.
  18. Girshick, R. (2015), "Fast r-cnn", Proceedings of the IEEE International Conference on Computer Vision.
  19. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  20. Guan, H., Li, J., Yu, Y., Chapman, M., Wang, H., Wang, C. and Zhai, R. (2014), "Iterative tensor voting for pavement crack extraction using mobile laser scanning data", IEEE Transact. Geosci. Remote Sens., 53(3), 1527-1537. https://doi.org/10.1109/TGRS.2014.2344714
  21. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 37(9), 1904-1916. https://doi.org/10.1109/TPAMI.2015.2389824
  22. Hu, J., Shen, L. and Sun, G. (2018), "Squeeze-and-excitation networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  23. Huyan, J., Li, W., Tighe, S., Zhai, J., Xu, Z. and Chen, Y. (2019), "Detection of sealed and unsealed cracks with complex backgrounds using deep convolutional neural network", Automat. Constr., 107, 102946. https://doi.org/10.1016/j.autcon.2019.102946
  24. Jang, K., Kim, N. and An, Y.-K. (2019), "Deep learning-based autonomous concrete crack evaluation through hybrid image scanning", Struct. Health Monitor., 18(5-6), 1722-1737. https://doi.org/10.1177/1475921718821719
  25. Jung, H.-J., Lee, J.H., Yoon, S.S. and Kim, I.H. (2019), "Bridge Inspection and condition assessment using Unmanned Aerial Vehicles (UAVs): Major challenges and solutions from a practical perspective", Smart Struct. Syst., Int. J., 24(5), 669-681. https://doi.org/10.12989/sss.2019.24.5.669
  26. Kang, D. and Cha, Y.J. (2018), "Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging", Comput.-Aided Civil Infrastr. Eng., 33(10), 885-902. https://doi.org/10.1111/mice.12375
  27. Kang, D., Benipal, S.S., Gopal, D.L. and Cha, Y.-J. (2020), "Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning", Automat. Constr., 118, 103291. https://doi.org/10.1016/j.autcon.2020.103291
  28. Kim, H., Lee, J., Ahn, E., Cho, S., Shin, M. and Sim, S.-H. (2017), "Concrete crack identification using a UAV incorporating hybrid image processing", Sensors, 17(9), 2052. https://doi.org/10.3390/s17092052
  29. Liu, Y. and Yeoh, J.K. (2021a), "Automated crack pattern recognition from images for condition assessment of concrete structures", Automat. Constr., 128, 103765. https://doi.org/10.1016/j.autcon.2021.103765
  30. Liu, Y. and Yeoh, J.K. (2021b), "Robust pixel-wise concrete crack segmentation and properties retrieval using image patches", Automat. Constr., 123, 103535. https://doi.org/10.1016/j.autcon.2020.103535
  31. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y. and Berg, A.C. (2016), "Ssd: Single shot multibox detector", In: European Conference on Computer Vision. https://doi.org/10.1007/978-3-319-46448-0_2
  32. Liu, Y.F., Nie, X., Fan, J.S. and Liu, X.G. (2020), "Image-based crack assessment of bridge piers using unmanned aerial vehicles and three-dimensional scene reconstruction", Comput.-Aided Civil Infrastr. Eng., 35(5), 511-529. https://doi.org/10.1111/mice.12501
  33. Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  34. Lu, J.Z., Hu, M.Q., Dong, J., Han, S.L. and Su, A. (2020), "A novel dense descriptor based on structure tensor voting for multi-modal image matching", Chinese J. Aeronaut., 33(9), 2408-2419. https://doi.org/10.1016/j.cja.2020.02.002
  35. Mordohai, P. and Medioni, G. (2006), "Tensor voting: a perceptual organization approach to computer vision and machine learning", In: Synthesis Lectures on Image, Video, and Multimedia Processing, 2(1), 1-136. https://doi.org/10.1007/978-3-031-02242-5
  36. MOT (2011), Standards for technical condition evaluation of highway bridges, MOT Beijing, China.
  37. Park, S.E., Eem, S.-H. and Jeon, H. (2020), "Concrete crack detection and quantification using deep learning and structured light", Constr. Build. Mater., 252, 119096. https://doi.org/10.1016/j.conbuildmat.2020.119096
  38. Peng, X., Zhong, X., Chen, A., Zhao, C., Liu, C. and Chen, Y.F. (2021a), "Debonding defect quantification method of building decoration layers via UAV-thermography and deep learning", Smart Struct. Syst., Int. J., 28(1), 55-67. https://doi.org/10.12989/sss.2021.28.1.055
  39. Peng, X., Zhong, X., Zhao, C., Chen, A. and Zhang, T. (2021b), "A UAV-based machine vision method for bridge crack recognition and width quantification through hybrid feature learning", Constr. Build. Mater., 299, 123896. https://doi.org/10.1016/j.conbuildmat.2021.123896
  40. Qin, J., Li, H., Xiang, X., Tan, Y., Pan, W., Ma, W. and Xiong, N.N. (2019), "An encrypted image retrieval method based on Harris corner optimization and LSH in cloud computing", IEEE Access, 7, 24626-24633. https://doi.org/10.1109/ACCESS.2019.2894673
  41. Redmon, J. and Farhadi, A. (2018), "Yolov3: An incremental improvement", arXiv preprint arXiv:1804.02767.
  42. Ribeiro, D., Santos, R., Shibasaki, A., Montenegro, P., Carvalho, H. and Calcada, R. (2020), "Remote inspection of RC structures using unmanned aerial vehicles and heuristic image processing", Eng. Fail. Anal., 117, 104813. https://doi.org/10.1016/j.engfailanal.2020.104813
  43. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October.
  44. Sterritt, G. (2009), Review of Bridge Inspection Competence and Training Project Rep, UK Bridges Board, London, UK.
  45. Wang, H.-F., Zhai, L., Huang, H., Guan, L.-M., Mu, K.-N. and Wang, G.-p. (2020), "Measurement for cracks at the bottom of bridges based on tethered creeping unmanned aerial vehicle", Automat. Constr., 119, 103330. https://doi.org/10.1016/j.autcon.2020.103330
  46. Yang, J., Fu, Q. and Nie, M. (2020), "Road crack detection using deep neural network with receptive field block", In: IOP Conference Series: Materials Science and Engineering.
  47. Yu, Z., Shen, Y. and Shen, C. (2021), "A real-time detection approach for bridge cracks based on YOLOv4-FPM", Automat. Constr., 122, 103514. https://doi.org/10.1016/j.autcon.2020.103514
  48. Zhang, C., Shu, J., Shao, Y. and Zhao, W. (2022), "Automated generation of FE models of cracked RC beams based on 3D point clouds and 2D images", J. Civil Struct. Health Monitor, 12(1), 29-46. https://doi.org/10.1007/s13349-021-00525-5
  49. Zhao, H., Yang, D. and Yu, J. (2021), "3D target detection using dual domain attention and SIFT operator in indoor scenes", Visual Comput., 38, 3365-3774. https://doi.org/10.1007/s00371-021-02217-z
  50. Zheng, M., Lei, Z. and Zhang, K. (2020), "Intelligent detection of building cracks based on deep learning", Image Vision Comput., 103, 103987. https://doi.org/10.1016/j.imavis.2020.103987
  51. Zhong, X., Peng, X., Yan, S., Shen, M. and Zhai, Y. (2018), "Assessment of the feasibility of detecting concrete cracks in images acquired by unmanned aerial vehicles", Automat. Constr., 89, 49-57. https://doi.org/10.1016/j.autcon.2018.01.005
  52. Zhou, Q., Ding, S., Qing, G. and Hu, J. (2022), "UAV vision detection method for crane surface cracks based on Faster RCNN and image segmentation", J. Civil Struct. Health Monitor., 12, 845-855. https://doi.org/10.1007/s13349-022-00577-1