Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Semantic crack-image identification framework for steel structures using atrous convolution-based Deeplabv3+ Network

  • Ta, Quoc-Bao (Department of Ocean Eng., Pukyong National University) ;
  • Dang, Ngoc-Loi (Urban Infrastructure Faculty, Mien Tay Construction University) ;
  • Kim, Yoon-Chul (Department of Civil and Environmental Eng., Yonsei University) ;
  • Kam, Hyeon-Dong (Department of Ocean Eng., Pukyong National University) ;
  • Kim, Jeong-Tae (Department of Ocean Eng., Pukyong National University)
  • Received : 2021.08.11
  • Accepted : 2022.03.21
  • Published : 2022.07.25
⟨ Previous Next ⟩

Abstract

For steel structures, fatigue cracks are critical damage induced by long-term cycle loading and distortion effects. Vision-based crack detection can be a solution to ensure structural integrity and performance by continuous monitoring and non-destructive assessment. A critical issue is to distinguish cracks from other features in captured images which possibly consist of complex backgrounds such as handwritings and marks, which were made to record crack patterns and lengths during periodic visual inspections. This study presents a parametric study on image-based crack identification for orthotropic steel bridge decks using captured images with complicated backgrounds. Firstly, a framework for vision-based crack segmentation using the atrous convolution-based Deeplapv3+ network (ACDN) is designed. Secondly, features on crack images are labeled to build three databanks by consideration of objects in the backgrounds. Thirdly, evaluation metrics computed from the trained ACDN models are utilized to evaluate the effects of obstacles on crack detection results. Finally, various training parameters, including image sizes, hyper-parameters, and the number of training images, are optimized for the ACDN model of crack detection. The result demonstrated that fatigue cracks could be identified by the trained ACDN models, and the accuracy of the crack-detection result was improved by optimizing the training parameters. It enables the applicability of the vision-based technique for early detecting tiny fatigue cracks in steel structures.

Keywords

Acknowledgement

This work was supported by a grant (21CTAP-C163708-01) from Technology Advancement Research Program funded by Korea Agency for Infrastructure Technology Advancement (KAIA). The datasets used in this paper were granted by the committee of the 1st International Project Competition for Structural Health Monitoring (IPC-SHM 2020). The authors would like to thank for the opportunity provided by IPC-SHM 2020.

References

  1. Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer, B.F. and Li, H. (2021), "The 1st international project competition for structural health monitoring (IPC-SHM, 2020): a summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1177/14759217211006485
  2. Barbedo, J.G.A. (2018), "Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification", Comput. Electron. Agricult., 153, 46-53. https://doi.org/10.1016/j.compag.2018.08.013
  3. Bastani, A., Amindavar, H., Shamshirsaz, M. and Sepehry, N. (2011), "Identification of temperature variation and vibration disturbance in impedance-based structural health monitoring using piezoelectric sensor array method", Struct. Health Monitor., Int. J., 11(3), 305-314. https://doi.org/10.1177/1475921711427486
  4. Bhalla, S., Vittal, P.A. and Veljkovic, M. (2012), "Piezo-impedance transducers for residual fatigue life assessment of bolted steel joints", Struct. Health Monitor., Int. J., 11(6), 733-750. https://doi.org/10.1177/1475921712458708
  5. Campbell, L.E., Connor, R.J., Whitehead, J.M. and Washer, G.A. (2020), "Benchmark for evaluating performance in visual inspection of fatigue cracking in steel bridges", J. Bridge Eng., 25(1), 04019128. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001507
  6. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastruct. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  7. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2014), "Semantic image segmentation with deep convolutional nets and fully connected crfs", Computer Vision and Pattern Recognition, arXiv preprint arXiv:1412.7062. https://doi.org/10.48550/arXiv.1412.7062
  8. Chen, L.C., Papandreou, G., Schroff, F. and Adam, H. (2017), "Rethinking atrous convolution for semantic image segmentation", arXiv preprint arXiv:1706.05587. https://doi.org/10.48550/arXiv.1706.05587
  9. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2018a), "DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Machine Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184
  10. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018b), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV). https://doi.org/10.48550/arXiv.1802.02611
  11. Chollet, F. (2017), "Xception: Deep learning with depthwise separable convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.48550/arXiv.1610.02357
  12. Connor, R.J. (2012), "Manual for design, construction, and maintenance of orthotropic steel deck bridges", No. FHWA-IF-12-027; United States, Federal Highway Administration. https://rosap.ntl.bts.gov/view/dot/41395
  13. Csurka, G., Larlus, D., Perronnin, F. and Meylan, F. (2013), "What is a good evaluation measure for semantic segmentation?", In: Bmvc, Vol. 27, pp. 10-5244. http://dx.doi.org/10.5244/C.27.32
  14. Dellenbaugh, L., Kong, X., Al-Salih, H., Collins, W., Bennett, C., Li, J. and Sutley, E.J. (2020), "Development of a distortion-induced fatigue crack characterization methodology using digital image correlation", J. Bridge Eng., 25(9), 04020063. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001598
  15. Dhivya, J.J. and Ramaswami, M. (2018), "A perusal analysis on hybrid spectrum handoff schemes in cognitive radio networks", International Conference on Intelligent Systems Design and Applications, pp. 312-321. https://doi.org/10.1007/978-3-030-16660-1_31
  16. Di, J., Ruan, X., Zhou, X., Wang, J. and Peng, X. (2021), "Fatigue assessment of orthotropic steel bridge decks based on strain monitoring data", Eng. Struct., 228, 111437. https://doi.org/10.1016/j.engstruct.2020.111437
  17. Dong, C.Z. and Catbas, F.N. (2020), "A review of computer vision-based structural health monitoring at local and global levels", Struct. Health Monitor., 20(2), 692-743. https://doi.org/10.1177/1475921720935585
  18. Dong, C.Z., Bas, S. and Catbas, F.N. (2020), "Investigation of vibration serviceability of a footbridge using computer vision-based methods", Eng. Struct., 224, 111224. https://doi.org/10.1016/j.engstruct.2020.111224
  19. Dong, C., Li, L., Yan, J., Zhang, Z., Pan, H. and Catbas, F.N. (2021), "Pixel-level fatigue crack segmentation in large-scale images of steel structures using an encoder-decoder network", Sensors, 21(12), 4135. https://doi.org/10.3390/s21124135
  20. Dung, C.V. and Anh, L.D. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  21. Erdogan, Y.S. and Ada, M. (2020), "A computer-vision based vibration transducer scheme for structural health monitoring applications", Smart Mater. Struct., 29(8), 085007. https://doi.org/10.1088/1361-665X/ab9062
  22. Fasl, J., Helwig, T. and Wood, S.L. (2016), "Fatigue response of a fracture-critical bridge at the end of service life", J. Perform. Constr. Facil, 30(5), 04016019. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000871
  23. Ghahremani, K., Sadhu, A., Walbridge, S. and Narasimhan, S. (2013), "Fatigue testing and structural health monitoring of retrofitted web stiffeners on steel highway bridges", Transport. Res. Record: J. Transport. Res. Board, 2360(1), 27-35. https://doi.org/10.3141/2360-04
  24. Habtour, E., Cole, D.P., Riddick, J.C., Weiss, V., Robeson, M., Sridharan, R. and Dasgupta, A. (2016), "Detection of fatigue damage precursor using a nonlinear vibration approach", Struct. Control Health Monitor., 23(12), 1442-1463. https://doi.org/10.1002/stc.1844
  25. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 37(9), 1904-1916. https://doi.org/10.1109/TPAMI.2015.2389824
  26. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2016.90
  27. Hou, Y., Yue, P., Xin, Q., Pauli, T., Sun, W. and Wang, L. (2013), "Fracture failure of asphalt binder in mixed mode (Modes I and II) by using phase-field model", Road Mater. Pave. Des., 15(1), 167-181. https://doi.org/10.1080/14680629.2013.866155
  28. Huynh, T.C. (2021), "Vision-based autonomous bolt-looseness detection method for splice connections: Design, lab-scale evaluation, and field application", Automat. Constr, 124, 103591. https://doi.org/10.1016/j.autcon.2021.103591
  29. Huynh, T.C., Park, Y.H., Park, J.H., Hong, D.S. and Kim, J.T. (2015), "Effect of temperature variation on vibration monitoring of prestressed concrete girders", Shock Vib., 1-9. https://doi.org/10.1155/2015/741618
  30. Huynh, T.C., Dang, N.L. and Kim, J.T. (2017), "Advances and challenges in impedance-based structural health monitoring", Struct. Monitor. Maint., Int. J., 4(4), 301-329. https://doi.org/10.12989/smm.2017.4.4.301
  31. Huynh, T.C., Dang, N.L. and Kim, J.T. (2018), "PCA-based filtering of temperature effect on impedance monitoring in prestressed tendon anchorage", Smart Struct. Syst., Int. J., 22(1), 57-70. https://doi.org/10.12989/sss.2018.22.1.057
  32. Huynh, T.C., Park, J.H., Jung, H.J. and Kim, J.T. (2019), "Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing", Automat. Constr., 105, 102844. https://doi.org/10.1016/j.autcon.2019.102844
  33. Huynh, T.C., Nguyen, T.T., Kim, J.T., Ta, Q.B., Ho, D.D. and Phan, T.T.V. (2021), "Deep learning-based functional assessment of piezoelectric-based smart interface under various degradations", Smart Struct. Syst., Int. J., 28(1), 69-87. https://doi.org/10.12989/sss.2021.28.1.069
  34. Jeong, S., Kim, H., Lee, J. and Sim, S.H. (2020), "Automated wireless monitoring system for cable tension forces using deep learning", Struct. Health Monitor., 20(4), 1805-1821. https://doi.org/10.1177/1475921720935837
  35. Jin, S.S., Jeong, S., Sim, S.H., Seo, D.W. and Park, Y.S. (2021), "Fully automated peak-picking method for an autonomous stay-cable monitoring system in cable-stayed bridges", Automat. Constr., 126, 103628. https://doi.org/10.1016/j.autcon.2021.103628
  36. Khuc, T. and Catbas, F.N. (2016), "Computer vision-based displacement and vibration monitoring without using physical target on structures", Struct. Infrastr. Eng., 13(4), 505-516. https://doi.org/10.1080/15732479.2016.1164729
  37. Kim, H., Yoon, J. and Sim, S.H. (2020), "Automated bridge component recognition from point clouds using deep learning", Struct. Control Health Monitor, 27(9). https://doi.org/10.1002/stc.2591
  38. Kong, X., Li, J., Collins, W., Bennett, C., Laflamme, S. and Jo, H. (2018), "Sensing distortion-induced fatigue cracks in steel bridges with capacitive skin sensor arrays", Smart Mater. Struct, 27(11), 115008. https://doi.org/10.1088/1361-665X/aadbfb
  39. Lee, Y.F., Lu, Y. and Guan, R. (2020), "Nonlinear guided waves for fatigue crack evaluation in steel joints with digital image correlation validation", Smart Mater. Struct., 29(3), 035031. https://doi.org/10.1088/1361-665X/ab6fe7
  40. Li, M., Suzuki, Y., Hashimoto, K. and Sugiura, K. (2018), "Experimental study on fatigue resistance of rib-to-deck joint in orthotropic steel bridge deck", J. Bridge Eng., 23(2), 04017128. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001175
  41. Ly, C.D., Vo, T.H., Mondal, S., Park, S., Choi, J., Vu, T.T.H., Kim, C.S. and Oh, J. (2021), "Full-view in vivo skin and blood vessels profile segmentation in photoacoustic imaging based on deep learning", Photoacoustics, 25, 100310. https://doi.org/10.1016/j.pacs.2021.100310
  42. Papadimitriou, C., Fritzen, C.P., Kraemer, P. and Ntotsios, E. (2011), "Fatigue predictions in entire body of metallic structures from a limited number of vibration sensors using Kalman filtering", Struct. Control Health Monitor., 18(5), 554-573. https://doi.org/10.1002/stc.395
  43. Park, J.-H., Huynh, T.-C., Choi, S.-H. and Kim, J.-T. (2015), "Vision-based technique for bolt-loosening detection in wind turbine tower", Wind Struct., Int. J., 21(6), 709-726. https://doi.org/10.12989/was.2015.21.6.709
  44. Pham, H.C., Ta, Q.B., Kim, J.T., Ho, D.D., Tran, X.L. and Huynh, T.C. (2020), "Bolt-loosening monitoring framework using an image-based deep learning and graphical model", Sensors, 20(12). https://doi.org/10.3390/s20123382
  45. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: convolutional networks for biomedical image segmentation", In: International Conference on Medical image computing and Computer-assisted Intervention, 9351. https://doi.org/10.1007/978-3-319-24574-4_28
  46. Shorten, C. and Khoshgoftaar, T.M. (2019), "A survey on image data augmentation for deep learning", J. Big Data, 6(1). https://doi.org/10.1186/s40537-019-0197-0
  47. Sim, H.B. and Uang, C.M. (2012), "Stress analyses and parametric study on full-scale fatigue tests of rib-to-deck welded joints in steel orthotropic decks", J. Bridge Eng., 17(5), 765-773. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000307
  48. Soh, C.K. and Lim, Y.Y. (2009), "Detection and characterization of fatigue induced damage using electromechanical impedance technique", Adv. Mater. Res., 79-82, 2031-2034. https://doi.org/10.4028/www.scientific.net/AMR.79-82.2031
  49. Spencer, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  50. Sun, L.M., Zhang, W. and Nagarajaiah, S. (2019), "Bridge real-time damage identification method using inclination and strain measurements in the presence of temperature variation", J. Bridge Eng., 24(2), 04018111. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001325
  51. Sun, Y., Yang, Y., Yao, G., Wei, F. and Wong, M. (2021), "Autonomous crack and bughole detection for concrete surface image based on deep learning", IEEE Access, 9, 85709-85720. https://doi.org/10.1109/ACCESS.2021.3088292
  52. Ta, Q.B. and Kim, J.T. (2020), "Monitoring of corroded and loosened bolts in steel structures via deep learning and hough transforms", Sensors, 20(23). https://doi.org/10.3390/s20236888
  53. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2018a), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumerg-rade camera images", Struct. Health Monitor, 18(3), 653-674. https://doi.org/10.1177/1475921718764873
  54. Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2018b), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monitor., 25(2), 2075. https://doi.org/10.1002/stc.2075
  55. Ya, S., Yamada, K. and Ishikawa, T. (2011), "Fatigue evaluation of rib-to-deck welded joints of orthotropic steel bridge deck", J. Bridge Eng., 16(4), 492-499. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000181
  56. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  57. Yao, L., Dong, Q., Jiang, J. and Ni, F. (2020), "Deep reinforcement learning for long-term pavement maintenance planning", Comput.-Aided Civil Infrastr. Eng., 35(11), 1230-1245. https://doi.org/10.1111/mice.12558
  58. Ye, X., Jin, T. and Yun, C. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst.., Int. J., 24(5), 567-586. https://doi.org/10.12989/sss.2019.24.5.567
  59. Zhao, X., Zhang, Y. and Wang, N. (2019), "Bolt loosening angle detection technology using deep learning", Struct. Control Health Monitor., 26(1), 2292. https://doi.org/10.1002/stc.2292