Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Crack segmentation in high-resolution images using cascaded deep convolutional neural networks and Bayesian data fusion

  • Tang, Wen (Lyles School of Civil Engineering, Purdue University) ;
  • Wu, Rih-Teng (Department of Civil Engineering, National Taiwan University) ;
  • Jahanshahi, Mohammad R. (Lyles School of Civil Engineering, Purdue University)
  • Received : 2021.05.08
  • Accepted : 2021.09.09
  • Published : 2022.01.25
⟨ Previous Next ⟩

Abstract

Manual inspection of steel box girders on long span bridges is time-consuming and labor-intensive. The quality of inspection relies on the subjective judgements of the inspectors. This study proposes an automated approach to detect and segment cracks in high-resolution images. An end-to-end cascaded framework is proposed to first detect the existence of cracks using a deep convolutional neural network (CNN) and then segment the crack using a modified U-Net encoder-decoder architecture. A Naïve Bayes data fusion scheme is proposed to reduce the false positives and false negatives effectively. To generate the binary crack mask, first, the original images are divided into 448 × 448 overlapping image patches where these image patches are classified as cracks versus non-cracks using a deep CNN. Next, a modified U-Net is trained from scratch using only the crack patches for segmentation. A customized loss function that consists of binary cross entropy loss and the Dice loss is introduced to enhance the segmentation performance. Additionally, a Naïve Bayes fusion strategy is employed to integrate the crack score maps from different overlapping crack patches and to decide whether a pixel is crack or not. Comprehensive experiments have demonstrated that the proposed approach achieves an 81.71% mean intersection over union (mIoU) score across 5 different training/test splits, which is 7.29% higher than the baseline reference implemented with the original U-Net.

Keywords

Acknowledgement

The authors would like to thank the organization of the IPC-SHM 2020: Harbin Institute of Technology, University of Illinois at Urbana-Champaign and ANCRiSST for providing the valuable data used in this study.

References

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
  2. Badrinarayanan, V., Kendall, A. and Cipolla, R. (2017), "Segnet: A deep convolutional encoder-decoder architecture for image segmentation", IEEE Transact. Pattern Anal. Mach. Intell., 39(12), 2481-2495. https://doi.org/10.1109/TPAMI.2016.2644615
  3. Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in black-box images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440
  4. Bao, Y. and Li, H. (2021), "Machine learning paradigm for structural health monitoring", Struct. Health Monitor., 20(4), 1353-1372. https://doi.org/10.1177/1475921720972416
  5. Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  6. Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer Jr, B.F. and Li, H. (2021), "The 1st International Project Competition for Structural Health Monitoring (IPC-SHM, 2020): A summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1177/14759217211006485
  7. Beckman, G.H., Polyzois, D. and Cha, Y.J. (2019), "Deep learning-based automatic volumetric damage quantification using depth camera", Automat. Constr., 99, 114-124. https://doi.org/10.1016/j.autcon.2018.12.006
  8. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  9. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  10. Chen, F.C. and Jahanshahi, M.R. (2017), "NB-CNN: Deep learning-based crack detection using convolutional neural network and Naive Bayes data fusion", IEEE Transact. Indust. Electron., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  11. Chen, F.C. and Jahanshahi, M.R. (2019), "NB-FCN: Real-time accurate crack detection in inspection videos using deep fully convolutional network and parametric data fusion", IEEE Transact. Instrument. Measur., 69(8), 5325-5334. https://doi.org/10.1109/TIM.2019.2959292
  12. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2017a), "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE Transact. Pattern Anal. Mach. Intell., 40(4), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184
  13. Chen, L.C., Papandreou, G., Schroff, F. and Adam, H. (2017b), "Rethinking atrous convolution for semantic image segmentation", arXiv. https://arxiv.org/abs/1706.05587
  14. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 801-818.
  15. Cheng, H.D., Chen, J.R., Glazier, C. and Hu, Y.G. (1999), "Novel approach to pavement cracking detection based on fuzzy set theory", J. Comput. Civil Eng., 13(4), 270-280. https://doi.org/10.1061/(ASCE)0887-3801(1999)13:4(270)
  16. Cheng, H.D., Shi, X.J. and Glazier, C. (2003), "Real-time image thresholding based on sample space reduction and interpolation approach", J. Comput. Civil Eng., 17(4), 264-272. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(264)
  17. Choi, W. and Cha, Y.J. (2019), "SDDNet: Real-time crack segmentation", IEEE Transact. Industr. Electron., 67(9), 8016-8025. https://doi.org/10.1109/TIE.2019.2945265
  18. Ciregan, D., Meier, U. and Schmidhuber, J. (2012), "Multi-column deep neural networks for image classification", IEEE Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, USA, June, pp. 3642-3649.
  19. Deng, J., Lu, Y. and Lee, V.C.S. (2020), "Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(4), 373-388. https://doi.org/10.1111/mice.12497
  20. Ding, L., Zhang, J. and Bruzzone, L. (2020), "Semantic segmentation of large-size VHR remote sensing images using a two-stage multiscale training architecture", IEEE Transact. Geosci. Remote Sensing, 58(8), 5367-5376. https://doi.org/10.1109/TGRS.2020.2964675
  21. Dumoulin, V. and Visin, F. (2016), "A guide to convolution arithmetic for deep learning", arXiv preprint. https://arxiv.org/abs/1603.07285
  22. Dung, C.V. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Automat. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  23. Fujita, Y. and Hamamoto, Y. (2011), "A robust automatic crack detection method from noisy concrete surfaces", Mach. Vis. Applicat., 22(2), 245-254. https://doi.org/10.1007/s00138-009-0244-5
  24. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", Proceedings of the IEEE International Conference on Computer Vision, Boston, MA, USA, June, pp. 1026-1034.
  25. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NA, USA, June, pp. 770-778.
  26. Ioffe, S. and Szegedy, C. (2015), "Batch normalization: Accelerating deep network training by reducing internal covariate shift", Proceedings of International Conference on Machine Learning, Lille, France, June, pp. 448-456.
  27. Jahanshahi, M.R., Kelly, J.S., Masri, S.F. and Sukhatme, G.S. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastr. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930
  28. Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vis. Applicat., 24(2), 227-241. https://doi.org/10.1007/s00138-011-0394-0
  29. Ji, A., Xue, X., Wang, Y., Luo, X. and Xue, W. (2020), "An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement", Automat. Constr., 114, 103176. https://doi.org/10.1016/j.autcon.2020.103176
  30. Kingma, D.P. and Ba, J. (2014), "Adam: A method for stochastic optimization", arXiv preprint. https://arxiv.org/abs/1412.6980
  31. LeCun, Y., Bengio, Y. and Hinton, G. (2015), "Deep learning", Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539
  32. Lee, B.J., Shin, D.H., Seo, J.W., Jung, J.D. and Lee, J.Y. (2011), "Intelligent bridge inspection using remote controlled robot and image processing technique", International Symposium on Automation and Robotics in Construction, Seoul, Korea, June.
  33. Li, X., Chen, S., Hu, X. and Yang, J. (2019), "Understanding the disharmony between dropout and batch normalization by variance shift", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June, pp. 2682-2690.
  34. Lim, R.S., La, H.M. and Sheng, W. (2014), "A robotic crack inspection and mapping system for bridge deck maintenance", IEEE Transact. Automat. Sci. Eng., 11(2), 367-378. https://doi.org/10.1109/TASE.2013.2294687
  35. Lin, M., Chen, Q. and Yan, S. (2013), "Network in network", arXiv preprint. https://arxiv.org/abs/1312.4400
  36. Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2017), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, October, pp. 2980-2988.
  37. Liu, Y., Ren, Q., Geng, J., Ding, M. and Li, J. (2018), "Efficient patch-wise semantic segmentation for large-scale remote sensing images", Sensors, 18(10), 3232. https://doi.org/10.3390/s18103232
  38. Liu, Y., Yao, J., Lu, X., Xie, R. and Li, L. (2019a), "DeepCrack: A deep hierarchical feature learning architecture for crack segmentation", Neurocomputing, 338, 139-153. https://doi.org/10.1016/j.neucom.2019.01.036
  39. Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019b), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005
  40. Liu, J., Yang, X., Lau, S., Wang, X., Luo, S., Lee, V.C.S. and Ding, L. (2020), "Automated pavement crack detection and segmentation based on two-step convolutional neural network", Comput.-Aided Civil Infrastr. Eng., 35(11), 1291-1305. https://doi.org/10.1111/mice.12622
  41. Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmentation", Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, June, pp. 3431-3440.
  42. Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018), "Road damage detection and classification using deep neural networks with smartphone images", Comput.-Aided Civil Infrastr. Eng., 33(12), 1127-1141. https://doi.org/10.1111/mice.12387
  43. Nair, V. and Hinton, G.E. (2010), "Rectified linear units improve restricted boltzmann machines", Proceedings of International Conference on Machine Learning, Haifa, Israel, June.
  44. Oh, J.K., Jang, G., Oh, S., Lee, J.H., Yi, B.J., Moon, Y.S., Lee, J.S. and Choi, Y. (2009), "Bridge inspection robot system with machine vision", Automat. Constr., 18(7), 929-941. https://doi.org/10.1016/j.autcon.2009.04.003
  45. Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster r-cnn: Towards real-time object detection with region proposal networks", Adv. Neural Inform. Process. Syst., 28, 91-99. https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
  46. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: Convolutional networks for biomedical image segmentation", International Conference on Medical Image Computing and Computer-Assisted Intervention, October, pp. 234-241.
  47. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  48. Tasar, O., Tarabalka, Y. and Alliez, P. (2019), "Incremental learning for semantic segmentation of large-scale remote sensing data", IEEE J. Select. Topics Appl. Earth Observ. Remote Sensing, 12(9), 3524-3537. https://doi.org/10.1109/JSTARS.2019.2925416
  49. Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2018), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monitor., 25(2), e2075. https://doi.org/10.1002/stc.2075
  50. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2019), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images", Struct. Health Monitor., 18(3), 653-674. https://doi.org/10.1177/1475921718764873
  51. Xue, Y. and Li, Y. (2018), "A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects", Comput.-Aided Civil Infrastr. Eng., 33(8), 638-654. https://doi.org/10.1111/mice.12367
  52. Yamaguchi, T. and Hashimoto, S. (2010), "Fast method for crack detection surface concrete large-size images using percolation-based image processing", Mach. Vis. Appl., 21, 797-809. https://doi.org/10.1007/s00138-009-0189-8
  53. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  54. Zhang, X., Rajan, D. and Story, B. (2019), "Concrete crack detection using context-aware deep semantic segmentation network", Comput.-Aided Civil Infrastr. Eng, 34(11), 951-971. https://doi.org/10.1111/mice.12477