Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Automatic assessment of post-earthquake buildings based on multi-task deep learning with auxiliary tasks

  • Zhihang Li (Department of Civil Engineering, Monash University) ;
  • Huamei Zhu (Department of Civil Engineering, Monash University) ;
  • Mengqi Huang (Department of Civil Engineering, Monash University) ;
  • Pengxuan Ji (Department of Civil Engineering, Monash University) ;
  • Hongyu Huang (Institute of Geotechnical Engineering, Zhejiang University) ;
  • Qianbing Zhang (Department of Civil Engineering, Monash University)
  • Received : 2022.09.07
  • Accepted : 2023.02.02
  • Published : 2023.04.25
⟨ Previous Next ⟩

Abstract

Post-earthquake building condition assessment is crucial for subsequent rescue and remediation and can be automated by emerging computer vision and deep learning technologies. This study is based on an endeavour for the 2nd International Competition of Structural Health Monitoring (IC-SHM 2021). The task package includes five image segmentation objectives - defects (crack/spall/rebar exposure), structural component, and damage state. The structural component and damage state tasks are identified as the priority that can form actionable decisions. A multi-task Convolutional Neural Network (CNN) is proposed to conduct the two major tasks simultaneously. The rest 3 sub-tasks (spall/crack/rebar exposure) were incorporated as auxiliary tasks. By synchronously learning defect information (spall/crack/rebar exposure), the multi-task CNN model outperforms the counterpart single-task models in recognizing structural components and estimating damage states. Particularly, the pixel-level damage state estimation witnesses a mIoU (mean intersection over union) improvement from 0.5855 to 0.6374. For the defect detection tasks, rebar exposure is omitted due to the extremely biased sample distribution. The segmentations of crack and spall are automated by single-task U-Net but with extra efforts to resample the provided data. The segmentation of small objects (spall and crack) benefits from the resampling method, with a substantial IoU increment of nearly 10%.

Keywords

Acknowledgement

This study was supported by Monash University for the scholarships and the high-performance computation platform sponsored by the 2022 AWS Cloud Computing Interdisciplinary Seed Project. The authors appreciate the organization committee of IC-SHM 2021, the University of Illinois at Urbana-Champaign, and the Harbin Institute of Technology, for generously providing the invaluable data. The authors also would like to thank the chairs of IC-SHM 2021, Prof. Billie F. Spencer Jr. and Prof. Hui Li, for leading this competition.

References

  1. Ando, R. and Zhang, T. (2005), "A framework for learning predictive structures from multiple tasks and unlabeled data.", J. Mach. Learn. Res., 6, 1817-1853.
  2. Azimi, M., Eslamlou, A.D. and Pekcan, G. (2020), "Data-driven structural health monitoring and damage detection through deep learning: State-of-the-art review", Sensors, 20(10), 2778. https://doi.org/10.3390/s20102778
  3. Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring" Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  4. Baxter, J. (1997), "A Bayesian/information theoretic model of learning to learn via multiple task sampling", Mach. Learn., 28(1), 7-39. https://doi.org/10.1023/A:1007327622663
  5. Blaikie, P., Cannon, T., Davis, I. and Wisner, B. (2014), At risk: Natural Hazards, People's Vulnerability and Disasters, Routledge.
  6. Cha, Y.-J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  7. Cha, Y.-J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  8. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2016), "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Mach. Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184 Retrieved from: https://ui.adsabs.harvard.edu/abs/2016arXiv160600915C
  9. Choi, W. and Cha, Y.-J. (2020), "SDDNet: Real-time crack segmentation", IEEE Transact. Indust. Electron., 67(9), 8016-8025. https://doi.org/10.1109/tie.2019.2945265
  10. Doocy, S., Daniels, A., Packer, C., Dick, A. and Kirsch, T.D. (2013), "The human impact of earthquakes: a historical review of events 1980-2009 and systematic literature review", PLoS currents, 5. https://doi.org/10.1371/currents.dis.67bd14fe457f1db0b5433a8ee20fb833
  11. Gao, Y. and Mosalam, K.M. (2018), "Deep transfer learning for image-based structural damage recognition", Comput.-Aided Civil Infrastr. Eng., 33(9), 748-768. https://doi.org/10.1111/mice.12363
  12. Guo, J., Wang, Q., Li, Y. and Liu, P. (2020), "Facade defects classification from imbalanced dataset using meta learning-based convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(12), 1403-1418. https://doi.org/10.1111/mice.12578
  13. He, K., Zhang, X., Ren, S. and Sun, J. (2014), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Transact. Pattern Anal. Mach. Intell., 37(9), 1904-1916. https://doi.org/10.1007/978-3-319-10578-9_23
  14. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  15. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B.F. (2018), "Towards automated post-earthquake inspections with deep learning-based condition-aware models", arXiv preprint arXiv:1809.09195. https://doi.org/10.48550/arXiv.1809.09195
  16. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B.F. (2020), "MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure", J. Civil Struct. Health Monitor., 10(5), 757-773. https://doi.org/10.1007/s13349-020-00409-0
  17. Hoskere, V., Narazaki, Y. and Spencer Jr, B.F. (2022), "Physics-based graphics models in 3D synthetic environments as autonomous vision-based inspection testbeds", Sensors, 22(2), 532. https://doi.org/10.3390/s22020532 Retrieved from: https://www.mdpi.com/1424-8220/22/2/532
  18. Lee, K., Lee, S. and Kim, H.Y. (2022), "Bounding-box object augmentation with random transformations for automated defect detection in residential building facades", Automat. Constr., 135. https://doi.org/10.1016/j.autcon.2022.104138
  19. Li, Y., Zhang, X. and Chen, D. (2018), "Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes", Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June, pp. 1091-1100.
  20. Li, J., Wang, Q., Ma, J. and Guo, J. (2022a), "Multi-defect segmentation from facade images using balanced copy-paste method", Comput.-Aided Civil Infrastr. Eng., 37(11), 1434-1449. https://doi.org/10.1111/mice.12808
  21. Li, Z., Huang, M., Ji, P., Zhu, H. and Zhang, Q. (2022b), "One-step deep learning-based method for pixel-level detection of fine cracks in steel girder images", Smart Struct. Syst., Int. J., 29(1), 153-166. https://doi.org/10.12989/sss.2022.29.1.153
  22. Mehdi, Z. and Nazmazar, B. (2013), "Van, Turkey earthquake of 23 october 2011, mw 7.2; an overview on disaster management", Iran. J. Public Health, 42(2), 134.
  23. Milletari, F., Navab, N. and Ahmadi, S.A. (2016), "V-net: Fully convolutional neural networks for volumetric medical image segmentation", Proceedings of the 4th International Conference on 3D Vision (3DV), pp. 565-571.
  24. Narazaki, Y., Hoskere, V., Hoang, T.A., Fujino, Y., Sakurai, A. and Spencer Jr, B.F. (2020), "Vision-based automated bridge component recognition with high-level scene consistency", Comput.-Aided Civil Infrastr. Eng., 35(5), 465-482. https://doi.org/10.1111/mice.12505
  25. Narazaki, Y., Hoskere, V., Yoshida, K., Spencer Jr, B.F. and Fujino, Y. (2021), "Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts", Mech. Syst. Signal Process., 160. https://doi.org/10.1016/j.ymssp.2021.107850
  26. Pan, X. and Yang, T.Y. (2020), "Postdisaster image-based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 35(5), 495-510. https://doi.org/10.1111/mice.12549
  27. Ruder, S. (2017), "An overview of multi-task learning in deep neural networks", arXiv:1706.05098. https://doi.org/10.48550/arXiv.1706.05098 Retrieved from: https://ui.adsabs.harvard.edu/abs/2017arXiv170605098R
  28. Sogaard, A. and Bingel, J. (2017), "Identifying beneficial task relations for multi-task learning in deep neural networks", arXiv preprint arXiv:1702.08303. https://doi.org/10.48550/arXiv.1702.08303
  29. Spencer Jr, B.F. and Li, H. (2021), The 2nd International Competition for Structural Health Monitoring (IC-SHM, 2021). Retrieved from: http://sstl.cee.illinois.edu/icshm2021/The%202nd%20International%20Project%20Competition%20-%20FINAL.pdf
  30. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  31. Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D. and Van Gool, L. (2020), "Multi-task learning for dense prediction tasks: A survey", IEEE Transact. Pattern Anal. Mach. Intell., 44(7), 3614-3633. https://doi.org/10.1109/TPAMI.2021.3054719Retrieved from: https://ui.adsabs.harvard.edu/abs/2020arXiv200413379V
  32. Yang, G., Li, G., Pan, T., Kong, Y., Wu, J., Shu, H., Luo, L., Dillenseger, J.L., Coatrieux, J.L., Tang, L. and Zhu, X. (2018), "Automatic segmentation of kidney and renal tumor in ct images based on 3d fully convolutional neural network with pyramid pooling module", Proceedings of the 24th International Conference on Pattern Recognition (ICPR), Beijing, China, August, pp. 3790-3795. https://doi.org/10.1109/ICPR.2018.8545143
  33. Yu, F. and Koltun, V. (2015), "Multi-scale context aggregation by dilated convolutions", arXiv preprint arXiv:1511.07122. https://doi.org/10.48550/arXiv.1511.07122
  34. Zhang, C.-H. and Huang, J. (2008), "The sparsity and bias of the Lasso selection in high-dimensional linear regression", Annals Statist., 36, 1567-1594. https://doi.org/10.1214/07-AOS520
  35. Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J. (2017), "Pyramid scene parsing network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 6230-6239.