Smart Structures and Systems

  • 제31권4호
  • /
  • Pages.365-381
  • /
  • 2023
  • /
  • 1738-1584(pISSN)
  • /
  • 1738-1991(eISSN)
테크노프레스 (Techno-Press)
권호 표지
DOI QR코드

DOI QR Code

Deep learning-based post-disaster building inspection with channel-wise attention and semi-supervised learning

  • Wen Tang (Lyles School of Civil Engineering, Purdue University) ;
  • Tarutal Ghosh Mondal (Department of Civil, Architecture and Environment Engineering, Missouri University of Science and Technology) ;
  • Rih-Teng Wu (Department of Civil Engineering, National Taiwan University) ;
  • Abhishek Subedi (Lyles School of Civil Engineering, Purdue University) ;
  • Mohammad R. Jahanshahi (Lyles School of Civil Engineering, Purdue University)
  • 투고 : 2022.09.18
  • 심사 : 2023.02.02
  • 발행 : 2023.04.25
⟨ 이전 논문 다음 논문 ⟩

초록

The existing vision-based techniques for inspection and condition assessment of civil infrastructure are mostly manual and consequently time-consuming, expensive, subjective, and risky. As a viable alternative, researchers in the past resorted to deep learning-based autonomous damage detection algorithms for expedited post-disaster reconnaissance of structures. Although a number of automatic damage detection algorithms have been proposed, the scarcity of labeled training data remains a major concern. To address this issue, this study proposed a semi-supervised learning (SSL) framework based on consistency regularization and cross-supervision. Image data from post-earthquake reconnaissance, that contains cracks, spalling, and exposed rebars are used to evaluate the proposed solution. Experiments are carried out under different data partition protocols, and it is shown that the proposed SSL method can make use of unlabeled images to enhance the segmentation performance when limited amount of ground truth labels are provided. This study also proposes DeepLab-AASPP and modified versions of U-Net++ based on channel-wise attention mechanism to better segment the components and damage areas from images of reinforced concrete buildings. The channel-wise attention mechanism can effectively improve the performance of the network by dynamically scaling the feature maps so that the networks can focus on more informative feature maps in the concatenation layer. The proposed DeepLab-AASPP achieves the best performance on component segmentation and damage state segmentation tasks with mIoU scores of 0.9850 and 0.7032, respectively. For crack, spalling, and rebar segmentation tasks, modified U-Net++ obtains the best performance with Igou scores (excluding the background pixels) of 0.5449, 0.9375, and 0.5018, respectively. The proposed architectures win the second place in IC-SHM2021 competition in all five tasks of Project 2.

키워드

과제정보

The authors would like to thank the organization of the IC-SHM 2021: ANCRiSST, University of Illinois at Urbana-Champaign, Harbin Institute of Technology, Zhejiang University, and University of Houston for providing the valuable data used in this study.

참고문헌

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Computer Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
  2. Bao, Y. and Li, H. (2021), "Machine learning paradigm for structural health monitoring", Struct. Health Monitor., 20(4), 1353-1372. https://doi.org/10.1177/1475921720972416
  3. Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  4. Beckman, G.H., Polyzois, D. and Cha, Y.J. (2019), "Deep learning-based automatic volumetric damage quantification using depth camera", Automat. Constr., 99, 114-124. https://doi.org/10.1016/j.autcon.2018.12.006
  5. Berman, M., Triki, A.R. and Blaschko, M.B. (2018), "The lovasz-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413-4421.
  6. Bermanmaxim. Bermanmaxim/LovaszSoftmax: Code for the lovasz-softmax loss (CVPR 2018), Retrieved January 28, 2022. https://github.com/bermanmaxim/LovaszSoftmax
  7. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  8. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  9. Chen, F.C. and Jahanshahi, M.R. (2017), "NB-CNN: Deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electron., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  10. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818.
  11. Chen, X., Yuan, Y., Zeng, G. and Wang, J. (2021), "Semi-supervised semantic segmentation with cross pseudo supervision", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2613-2622.
  12. Cheng, H.D., Chen, J.R., Glazier, C. and Hu, Y.G. (1999), "Novel approach to pavement cracking detection based on fuzzy set theory", J. Computer Civil Eng., 13(4), 270-280. https://doi.org/10.1061/(ASCE)0887-3801(1999)13:4(270)
  13. Cheng, H.D., Shi, X.J. and Glazier, C. (2003), "Real-time image thresholding based on sample space reduction and interpolation approach", J. Computer Civil Eng., 17(4), 264-272. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(264)
  14. Choi, W. and Cha, Y.J. (2019), "SDDNet: Real-time crack segmentation", IEEE Transact. Industr. Electron., 67(9), 8016-8025. https://doi.org/10.1109/TIE.2019.2945265
  15. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K. and Fei-Fei, L. (2009), "Imagenet: A large-scale hierarchical image database", Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, June, pp. 248-255. https://doi.org/10.1109/CVPR.2009.5206848
  16. Dong, C., Li, L., Yan, J., Zhang, Z., Pan, H. and Catbas, F.N. (2021), "Pixel-level fatigue crack segmentation in large-scale images of steel structures using an encoder-decoder network", Sensors, 21(12), 4135. https://doi.org/10.3390/s21124135
  17. Dung, C.V. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  18. Fujita, Y. and Hamamoto, Y. (2011), "A robust automatic crack detection method from noisy concrete surfaces", Mach. Vis. Applicat., 22(2), 245-254. https://doi.org/10.1007/s00138-009-0244-5
  19. Gao, Y. and Mosalam, K.M. (2018), "Deep transfer learning for image-based structural damage recognition", Comput.-Aided Civil Infrastr. Eng., 33(9), 748-768. https://doi.org/10.1111/mice.12363
  20. Ghosh Mondal, T., Jahanshahi, M.R., Wu, R.T. and Wu, Z.Y. (2020), "Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance", Struct. Control Health Monitor., 27(4), e2507. https://doi.org/10.1002/stc.2507
  21. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 37(9), 1904-1916. https://doi.org/10.1109/TPAMI.2015.2389824
  22. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778.
  23. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B.F. (2020), "MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure", J. Civil Struct. Health Monitor., 10, 757-773. https://doi.org/10.1007/s13349-020-00409-0
  24. Hoskere, V., Narazaki, Y. and Spencer, B.F. (2022), "Physics-based graphics models in 3D synthetic environments as autonomous vision-based inspection testbeds", Sensors, 22(2), 532. https://doi.org/10.3390/s22020532
  25. Hu, J., Shen, L. and Sun, G. (2018), "Squeeze-and-excitation networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141.
  26. Ji, A., Xue, X., Wang, Y., Luo, X. and Xue, W. (2020), "An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement", Automat. Constr., 114, 103176. https://doi.org/10.1016/j.autcon.2020.103176
  27. Ke, Z., Qiu, D., Li, K., Yan, Q. and Lau, R.W. (2020), "Guided collaborative training for pixel-wise semi-supervised learning", Proceedings of European Conference on Computer Vision, pp. 429-445. https://doi.org/10.1007/978-3-030-58601-0_26
  28. Lau, S.L., Chong, E.K., Yang, X. and Wang, X. (2020), "Automated pavement crack segmentation using u-net-based convolutional neural network", IEEE Access, 8, 114892-114899. https://doi.org/10.1109/ACCESS.2020.3003638
  29. Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2017), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988.
  30. Liu, Y., Yao, J., Lu, X., Xie, R. and Li, L. (2019), "DeepCrack: A deep hierarchical feature learning architecture for crack segmentation", Neurocomputing, 338, 139-153. https://doi.org/10.1016/j.neucom.2019.01.036
  31. Liu, J., Yang, X., Lau, S., Wang, X., Luo, S., Lee, V.C.S. and Ding, L. (2020), "Automated pavement crack detection and segmentation based on two-step convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(11), 1291-1305. https://doi.org/10.1111/mice.12622
  32. Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June, pp. 3431-3440.
  33. Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018), "Road damage detection and classification using deep neural networks with smartphone images", Comput.-Aided Civil Infrastr. Eng., 33(12), 1127-1141. https://doi.org/10.1111/mice.12387
  34. Narazaki, Y., Hoskere, V., Eick, B.A., Smith, M.D. and Spencer, B.F. (2019), "Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models", Smart Struct. Syst., Int. J., 24(6), 709-721. https://doi.org/10.12989/sss.2019.24.6.709
  35. Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B.F. and Fujino, Y. (2021), "Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts", Mech. Syst. Signal Process., 160, 107850. https://doi.org/10.1016/j.ymssp.2021.107850
  36. Ouali, Y., Hudelot, C. and Tami, M. (2020), "Semi-supervised semantic segmentation with cross-consistency training", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12674-12684.
  37. Paszke, A., Chaurasia, A., Kim, S. and Culurciello, E. (2016), "ENet: A deep neural network architecture for real-time semantic segmentation", arXiv preprint arXiv:1606.02147. https://doi.org/10.48550/arXiv.1606.02147
  38. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, October, pp. 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
  39. Sajedi, S.O. and Liang, X. (2021), "Uncertainty-assisted deep vision structural health monitoring", Comput.-Aided Civil Infrastr. Eng., 36(2), 126-142. https://doi.org/10.1111/mice.12580
  40. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  41. Tang, W., Wu, R.-T. and Jahanshahi, M.R. (2022), "Crack segmentation in high-resolution images using cascaded deep convolutional neural networks and Bayesian data fusion", Smart Struct. Syst., Int. J., 29(1), 221-235. https://doi.org/10.12989/sss.2022.29.1.221
  42. Xue, Y. and Li, Y. (2018), "A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects", Comput.-Aided Civil Infrastr. Eng., 33(8), 638-654. https://doi.org/10.1111/mice.12367
  43. Yamaguchi, T. and Hashimoto, S. (2010), "Fast method for crack detection surface concrete large-size images using percolation-based image processing", Mach. Vis. Appl., 21, 797-809. https://doi.org/10.1007/s00138-009-0189-8
  44. Yang, Q. and Ji, X. (2021), "Automatic pixel-level crack detection for civil infrastructure using UNet++ and deep transfer learning", IEEE Sensors J., 21(17), 19165-19175. https://doi.org/10.1109/JSEN.2021.3089718
  45. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  46. Zhang, X., Rajan, D. and Story, B. (2019), "Concrete crack detection using context-aware deep semantic segmentation network", Comput.-Aided Civil Infrastr. Eng., 34(11), 951-971. https://doi.org/10.1111/mice.12477
  47. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N. and Liang, J. (2018), "Unet++: A nested u-net architecture for medical image segmentation", In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 3-11.
  48. Zou, Y., Zhang, Z., Zhang, H., Li, C.L., Bian, X., Huang, J.B. and Pfister, T. (2020), "Pseudoseg: Designing pseudo labels for semantic segmentation", arXiv preprint arXiv:2010.09713. https://doi.org/10.48550/arXiv.2010.09713