Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A three-stage deep-learning-based method for crack detection of high-resolution steel box girder image

  • Meng, Shiqiao (State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University) ;
  • Gao, Zhiyuan (College of Civil Engineering, Tongji University) ;
  • Zhou, Ying (State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University) ;
  • He, Bin (College of Electronic and Information Engineering, Tongji University) ;
  • Kong, Qingzhao (State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University)
  • Received : 2021.03.27
  • Accepted : 2021.09.09
  • Published : 2022.01.25
⟨ Previous Next ⟩

Abstract

Crack detection plays an important role in the maintenance and protection of steel box girder of bridges. However, since the cracks only occupy an extremely small region of the high-resolution images captured from actual conditions, the existing methods cannot deal with this kind of image effectively. To solve this problem, this paper proposed a novel three-stage method based on deep learning technology and morphology operations. The training set and test set used in this paper are composed of 360 images (4928 × 3264 pixels) in steel girder box. The first stage of the proposed model converted high-resolution images into sub-images by using patch-based method and located the region of cracks by CBAM ResNet-50 model. The Recall reaches 0.95 on the test set. The second stage of our method uses the Attention U-Net model to get the accurate geometric edges of cracks based on results in the first stage. The IoU of the segmentation model implemented in this stage attains 0.48. In the third stage of the model, we remove the wrong-predicted isolated points in the predicted results through dilate operation and outlier elimination algorithm. The IoU of test set ascends to 0.70 after this stage. Ablation experiments are conducted to optimize the parameters and further promote the accuracy of the proposed method. The result shows that: (1) the best patch size of sub-images is 1024 × 1024. (2) the CBAM ResNet-50 and the Attention U-Net achieved the best results in the first and the second stage, respectively. (3) Pre-training the model of the first two stages can improve the IoU by 2.9%. In general, our method is of great significance for crack detection.

Keywords

Acknowledgement

The authors acknowledge the financial support from National Natural Science Foundation of China (Grant No. 52025083). The authors would like to thank the organizations of the International Project Competition for SHM (IPC-SHM 2020) ANCRiSST, Harbin Institute of Technology (China), and University of Illinois at Urbana-Champaign (USA) for their generously providing the invaluable data from actual structures. The authors also would like to thank the chairs of IPC-SHM 2020 Prof. Hui Li, and Prof. Billie F. Spencer Jr for their leadership on the competition.

References

  1. Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in black-box images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440
  2. Bao, Y. and Li, H. (2021), "Machine learning paradigm for structural health monitoring", Struct. Health Monitor., 20(4), 1353-1372. https://doi.org/10.1177/1475921720972416
  3. Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  4. Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer, B.F. and Li, H. (2021), "The 1st International Project Competition for Structural Health Monitoring (IPC-SHM, 2020): A summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1177/14759217211006485
  5. Canny, J. (1986), "A computational approach to edge detection", IEEE Transact. Pattern Anal. Mach. Intell., (6), 679-698.
  6. Cha, Y.-J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous Structural Visual Inspection Using Region-Based Deep Learning for Detecting Multiple Damage Types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  7. Chen, F.-C. and Jahanshahi, M.R. (2017), "NB-CNN: Deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electro., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  8. de Freitas, S.T., Kolstein, H. and Bijlaard, F. (2012), "Parametric study on the interface layer of renovation solutions for orthotropic steel bridge decks", Comput.-Aided Civil Infrastr. Eng., 27(2), 143-153. https://doi.org/10.1111/j.1467-8667.2010.00693.x
  9. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K. and Fei-Fei, L. (2009), "Imagenet: A large-scale hierarchical image database", Proceedings of the IEEE conference on computer vision and pattern recognition, Miami Beach, FL, USA, June.
  10. Dung, C.V. and Anh, L.D. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  11. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, June.
  12. Han, Q., Xu, J., Carpinteri, A. and Lacidogna, G. (2015), "Localization of acoustic emission sources in structural health monitoring of masonry bridge", Struct. Control Health Monitor., 22(2), 314-329. https://doi.org/10.1002/stc.1675
  13. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, VA, USA, June.
  14. Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E. and Saltz, J.H. (2016), "Patch-based con- volutional neural network for whole slide tissue image classification", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, VA, USA, June.
  15. Hu, J., Shen, L. and Sun, G. (2018), "Squeeze-and-excitation networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June.
  16. Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K.Q. (2017), "Densely connected convolutional networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.
  17. Kingma, D.P. and Ba, J. (2014), "Adam: A method for stochastic optimization", Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA, May.
  18. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2017), "ImageNet classification with deep convolutional neural networks", Commun. ACM, 60(6), 84-90. https://doi.org/10.1145/3065386
  19. Li, S., Zhao, X. and Zhou, G. (2019), "Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 34(7), 616-634. https://doi.org/10.1111/mice.12433
  20. Lim, R.S., La, H.M. and Sheng, W. (2014), "A robotic crack inspection and mapping system for bridge deck maintenance", IEEE Transact. Automat. Sci. Eng., 11(2), 367-378. https://doi.org/10.1109/TASE.2013.2294687
  21. Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmen- tation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June.
  22. Mutlib, N.K., Baharom, S.B., El-Shafie, A. and Nuawi, M.Z. (2016), "Ultrasonic health monitoring in structural engineering: buildings and bridges", Struct. Control Health Monitor., 23(3), 409-422. https://doi.org/10.1002/stc.1800
  23. Oh, J.-K., Jang, G., Oh, S., Lee, J.H., Yi, B.-J., Moon, Y.S., Lee, J.S. and Choi, Y. (2009). "Bridge inspection robot system with machine vision", Automat. Constr., 18(7), 929-941. https://doi.org/10.1016/j.autcon.2009.04.003
  24. Ong, E.P., Lee, J.A., Cheng, J., Xu, G., Lee, B.H., Laude, A., Teoh, S., Lim T.H., Wong D.W.K. and Liu, J. (2015), "A robust outlier elimination approach for multimodal retina image registration", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October.
  25. Pan, Y., Zhang, G. and Zhang, L. (2020). "A spatial-channel hierarchical deep learning network for pixel-level automated crack detection", Automat. Constr., 119, 103357. https://doi.org/10.1016/j.autcon.2020.103357
  26. Qiao, W., Liu, Q., Wu, X., Ma, B. and Li, G. (2021), "Automatic Pixel-Level Pavement Crack Recognition Using a Deep Feature Aggregation Segmentation Network with a scSE Attention Mechanism Module", Sensors, 21(9), 2902. https://doi.org/10.3390/s21092902
  27. Ren, S., He, K., Girshick, R. and Sun, J. (2016), "Faster R-CNN: Towards real-time object detection with region proposal networks", arXiv:1506.01497 [cs]. http://arxiv.org/abs/1506.01497
  28. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer- Assisted Intervention, Munich, Germany, October.
  29. Saouma, V.E., Barton, C.C. and Gamaleldin, N.A. (1990), "Fractal characterization of fracture surfaces in concrete", Eng. Fract. Mech., 35(1-3), 47-53. https://doi.org/10.1016/0013-7944(90)90182-G
  30. Sofia, T.D.F., Henk, K. and Frans, B. (2012), "Parametric study on the interface layer of renovation solutions for orthotropic steel bridge decks", Comput.-Aided Civil Infrastr. Eng., 27(2), 143-153. https://doi.org/10.1111/j.1467-8667.2010.00693.x
  31. Song, W., Jia, G., Jia, D. and Zhu, H. (2019), "Automatic Pavement Crack Detection and Classification Using Multiscale Feature Attention Network", IEEE Access, 7, 171001-171012. https://doi.org/10.1109/ACCESS.2019.2956191
  32. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Eng., 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  33. Sun, B.-C. and Qiu, Y. (2007). "Automatic identification of pavement cracks using mathematic morphology", Proceedings of International Conference on Transportation Engineering 2007, pp. 1783-1788.
  34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", arXiv preprint arXiv:1706.03762.
  35. Wan, H., Gao, L., Su, M., Sun, Q. and Huang, L. (2021), "Attention-Based Convolutional Neural Network for Pavement Crack Detection", Adv. Mater. Sci. Eng. https://doi.org/10.1155/2021/5520515
  36. Woo, S., Park, J., Lee, J.Y. and Kweon, I.S. (2018), "Cbam: Convolutional block attention module", Proceedings of the European Conference on Computer Vision, Munich, Germany, September.
  37. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2019), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images", Struct. Health Monitor., 18(3), 653-674. https://doi.org/10.1177/1475921718764873
  38. Yang, Y. and Nagarajaiah, S. (2014), "Blind identification of damage in time-varying systems using independent component analysis with wavelet transform", Mech. Syst. Signal Process., 47(1-2), 3-20. https://doi.org/10.1016/j.ymssp.2012.08.029
  39. Yang, Y. and Nagarajaiah, S. (2016), "Harnessing data structure for recovery of randomly missing structural vibration responses time history: Sparse representation versus low-rank structure", Mech. Syst. Signal Process., 74, 165-182. https://doi.org/10.1016/j.ymssp.2015.11.009
  40. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  41. Ye, X.-W., Jin, T. and Chen, P.-Y. (2019), "Structural crack detection using deep learning-based fully convolutional networks", Adv. Struct. Eng., 22(16), 3412-3419. https://doi.org/10.1177/1369433219836292
  42. Zhang, A., Wang, K.C.P., Fei, Y., Liu, Y., Tao, S., Chen, C., Li, J.Q. and Li, B. (2018), "Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet", J. Comput. Civil Eng., 32(5), 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775