Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A hierarchical semantic segmentation framework for computer vision-based bridge damage detection

  • Jingxiao Liu (Department of Civil and Environmental Engineering, Stanford University) ;
  • Yujie Wei (Department of Civil and Environmental Engineering, Carnegie Mellon University) ;
  • Bingqing Chen (Department of Civil and Environmental Engineering, Carnegie Mellon University) ;
  • Hae Young Noh (Department of Civil and Environmental Engineering, Stanford University)
  • Received : 2022.09.13
  • Accepted : 2023.02.02
  • Published : 2023.04.25
⟨ Previous Next ⟩

Abstract

Computer vision-based damage detection enables non-contact, efficient and low-cost bridge health monitoring, which reduces the need for labor-intensive manual inspection or that for a large number of on-site sensing instruments. By leveraging recent semantic segmentation approaches, we can detect regions of critical structural components and identify damages at pixel level on images. However, existing methods perform poorly when detecting small and thin damages (e.g., cracks); the problem is exacerbated by imbalanced samples. To this end, we incorporate domain knowledge to introduce a hierarchical semantic segmentation framework that imposes a hierarchical semantic relationship between component categories and damage types. For instance, certain types of concrete cracks are only present on bridge columns, and therefore the noncolumn region may be masked out when detecting such damages. In this way, the damage detection model focuses on extracting features from relevant structural components and avoid those from irrelevant regions. We also utilize multi-scale augmentation to preserve contextual information of each image, without losing the ability to handle small and/or thin damages. In addition, our framework employs an importance sampling, where images with rare components are sampled more often, to address sample imbalance. We evaluated our framework on a public synthetic dataset that consists of 2,000 railway bridges. Our framework achieves a 0.836 mean intersection over union (IoU) for structural component segmentation and a 0.483 mean IoU for damage segmentation. Our results have in total 5% and 18% improvements for the structural component segmentation and damage segmentation tasks, respectively, compared to the best-performing baseline model.

Keywords

Acknowledgement

This research was supported in part by the Leavell Fellowship on Sustainable Built Environment at Stanford University.

References

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255) 
  2. ASCE (2021), Bridges-infrastructure report card. 
  3. Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in black- box images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440 
  4. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263 
  5. Deng, J., Lu, Y. and Lee, V.C.S. (2020), "Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(4), 373-388. https://doi.org/10.1111/mice.12497 
  6. Dong, C.Z. and Catbas, F.N. (2021), "A review of computer vision-based structural health monitoring at local and global levels", Struct. Health Monitor., 20(2), 692-743. https://doi.org/10.1177/1475921720935585 
  7. Dung, C.V. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028 
  8. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B.F. (2018), "Towards automated post-earthquake inspections with deep learning-based condition-aware models", arXiv preprint arXiv:1809.09195. 
  9. Hartle, R.A., Ryan, T.W., Mann, E., Danovich, L.J., Sosko, W.B., Bouscher, J.W. and Baker Jr, M. (2002), "Bridge Inspector's Reference Manual: Volume 1 and Volume 2 (No. DTFH61-97-D-00025)", United States. Federal Highway Administration. 
  10. Katharopoulos, A. and Fleuret, F. (2018), "Not all samples are created equal: Deep learning with importance sampling", Proceedings of International Conference on Machine Learning, pp. 2525-2534. 
  11. Kim, B. and Cho, S. (2019), "Image-based concrete crack assessment using mask and region-based convolutional neural network", Struct. Control Health Monitor., 26(8), e2381. https://doi.org/10.1002/stc.2381 
  12. Li, L., Zhou, T., Wang, W., Li, J. and Yang, Y. (2022), "Deep Hierarchical Semantic Segmentation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1246-1257. 
  13. Liang, X. (2019), "Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization", Comput.-Aided Civil Infrastr. Eng., 34(5), 415-430. https://doi.org/10.1111/mice.12425 
  14. Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005 
  15. Liu, J., Chen, S., Berges, M., Bielak, J., Garrett, J.H., Kovacevic, J. and Noh, H.Y. (2020), "Diagnosis algorithms for indirect structural health monitoring of a bridge model via dimensionality reduction", Mech. Syst. Signal Process., 136, 106454. https://doi.org/10.1016/j.ymssp.2019.106454 
  16. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S. and Guo, B. (2021), "Swin transformer: Hierarchical vision transformer using shifted windows", Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022. 
  17. Liu, J., Xu, S., Berges, M. and Noh, H.Y. (2022), "HierMUD: Hierarchical multi-task unsupervised domain adaptation between bridges for drive-by damage diagnosis", Struct. Health Monitor, p. 14759217221081159. https://doi.org/10.1177/14759217221081159 
  18. Narazaki, Y., Hoskere, V., Yoshida, K., Spencer Jr., B.F. and Fujino, Y. (2021), "Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts", Mech. Syst. Signal Process., 160, 107850. https://doi.org/10.1016/j.ymssp.2021.107850 
  19. Noh, H., Hong, S. and Han, B. (2015), "Learning deconvolution network for semantic segmentation", Proceedings of the IEEE International Conference on Computer Vision, pp. 1520-1528. 
  20. Ren, Y., Huang, J., Hong, Z., Lu, W., Yin, J., Zou, L. and Shen, X. (2020), "Image-based concrete crack detection in tunnels using deep fully convolutional networks", Constr. Build. Mater., 234, 117367. https://doi.org/10.1016/j.conbuildmat.2019.117367 
  21. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. 
  22. Shelhamer, E., Long, J. and Darrell, T. (2017). "Fully convolutional networks for semantic segmentation", IEEE Trans. Pattern Anal. Mach. Intell., 39(4), 640-651.  https://doi.org/10.1109/TPAMI.2016.2572683
  23. Shrivastava, A., Gupta, A. and Girshick, R. (2016), "Training region-based object detectors with online hard example mining", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 761-769. 
  24. Sony, S., Laventure, S. and Sadhu, A. (2019), "A literature review of next-generation smart sensing technology in structural health monitoring", Struct. Control Health Monitor., 26(3), e2321. https://doi.org/10.1002/stc.2321 
  25. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030 
  26. Sun, L., Shang, Z., Xia, Y., Bhowmick, S. and Nagarajaiah, S. (2020), "Review of bridge structural health monitoring aided by big data and artificial intelligence: From condition assessment to damage detection", J. Struct. Eng., 146(5), 04020073. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002535 
  27. Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X. and Liu, W. (2020), "Deep highresolution representation learning for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 43(10), 3349-3364. https://doi.org/10.1109/TPAMI.2020.2983686 
  28. Xiao, T., Liu, Y., Zhou, B., Jiang, Y. and Sun, J. (2018), "Unified perceptual parsing for scene understanding", Proceedings of the European conference on computer vision (ECCV), pp. 418-434. 
  29. Xu, H., Su, X., Wang, Y., Cai, H., Cui, K. and Chen, X. (2019), "Automatic bridge crack detection using a convolutional neural network", Appl. Sci., 9(14), 2867. https://doi.org/10.3390/app9142867 
  30. Yuan, Y., Chen, X. and Wang, J. (2020), "Object-contextual representations for semantic segmentation", Proceedings of European Conference on Computer Vision, pp. 173-190. 
  31. Zhang, J., Lu, C., Wang, J., Wang, L. and Yue, X.G. (2019), "Concrete cracks detection based on FCN with dilated convolution", Appl. Sci., 9(13), 2686. https://doi.org/10.3390/app9132686 
  32. Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R. and Li, M. (2022), "Resnest: Split-attention networks", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2736-2746. 
  33. Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J. (2017), "Pyramid scene parsing network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881-2890. 
  34. Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A. and Torralba, A. (2019), "Semantic understanding of scenes through the ade20k dataset", Int. J. Comput. Vision, 127(3), 302-321. https://doi.org/10.1007/s11263-018-1140-0