Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A modified U-net for crack segmentation by Self-Attention-Self-Adaption neuron and random elastic deformation

  • Zhao, Jin (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Hu, Fangqiao (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Qiao, Weidong (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Zhai, Weida (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Xu, Yang (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Bao, Yuequan (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology) ;
  • Li, Hui (Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology)
  • Received : 2021.03.06
  • Accepted : 2021.06.01
  • Published : 2022.01.25
⟨ Previous Next ⟩

Abstract

Despite recent breakthroughs in deep learning and computer vision fields, the pixel-wise identification of tiny objects in high-resolution images with complex disturbances remains challenging. This study proposes a modified U-net for tiny crack segmentation in real-world steel-box-girder bridges. The modified U-net adopts the common U-net framework and a novel Self-Attention-Self-Adaption (SASA) neuron as the fundamental computing element. The Self-Attention module applies softmax and gate operations to obtain the attention vector. It enables the neuron to focus on the most significant receptive fields when processing large-scale feature maps. The Self-Adaption module consists of a multiplayer perceptron subnet and achieves deeper feature extraction inside a single neuron. For data augmentation, a grid-based crack random elastic deformation (CRED) algorithm is designed to enrich the diversities and irregular shapes of distributed cracks. Grid-based uniform control nodes are first set on both input images and binary labels, random offsets are then employed on these control nodes, and bilinear interpolation is performed for the rest pixels. The proposed SASA neuron and CRED algorithm are simultaneously deployed to train the modified U-net. 200 raw images with a high resolution of 4928 × 3264 are collected, 160 for training and the rest 40 for the test. 512 × 512 patches are generated from the original images by a sliding window with an overlap of 256 as inputs. Results show that the average IoU between the recognized and ground-truth cracks reaches 0.409, which is 29.8% higher than the regular U-net. A five-fold cross-validation study is performed to verify that the proposed method is robust to different training and test images. Ablation experiments further demonstrate the effectiveness of the proposed SASA neuron and CRED algorithm. Promotions of the average IoU individually utilizing the SASA and CRED module add up to the final promotion of the full model, indicating that the SASA and CRED modules contribute to the different stages of model and data in the training process.

Keywords

Acknowledgement

Financial support for this study was provided by the National Natural Science Foundation of China [Grant Nos. 52008138, 51638007, U1711265, and 51921006], National Key R&D Program of China [Grant No. 2019YFC1511102], China Postdoctoral Science Foundation [Grant Nos. BX20190102 and 2019M661286], and Heilongjiang Postdoctoral Funding [Grant Nos. LBH-TZ2016 and LBH-Z19064].

References

  1. Abdel-Qader, I., Abudayyeh, O. and Kelly, M.E. (2003), "Analysis of edge-detection techniques for crack identification in bridges", J. Comput. Civil Eng., 17(4), 255-263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255)
  2. Agrawal, A. and Mittal, N. (2020), "Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy", Visual Comput., 36(2), 405-412. https://doi.org/10.1007/s00371-019-01630-9
  3. Badrinarayanan, V., Kendall, A. and Cipolla, R. (2017), "Segnet: a deep convolutional encoder-decoder architecture for image segmentation", IEEE Transact. Pattern Anal. Mach. Intell., 39(12), 2481-2495. https://doi.org/10.1109/TPAMI.2016.2644615
  4. Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in black-box images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440
  5. Bao, Y. and Li, H. (2020), "Machine learning paradigm for structural health monitoring", Struct. Health Monitor., 1475921720972416. https://doi.org/10.1177/1475921720972416
  6. Bao, Y., Chen, Z., Wei, S., Tang, Z., Xu, Y. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Eng., 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  7. Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer, B.F. Jr. and Li, H. (2021), "The 1st international project competition for structural health monitoring (IPC-SHM, 2020): a summary and benchmark problem", Struct. Health Monitor., 14759217211006485. https://doi.org/10.1177/14759217211006485
  8. Beckman, G.H., Polyzois, D. and Cha, Y. (2019), "Deep learning-based automatic volumetric damage quantification using depth camera", Automat. Constr., 99, 114-124. https://doi.org/10.1016/j.autcon.2018.12.006
  9. Bengio, Y. (2012), "Practical recommendations for gradient-based training of deep architectures", Neural Networks: Tricks of the Trade, Springer, Berlin, Heidelberg.
  10. Benz, C., Debus, P., Ha, H.K. and Rodehorst, V. (2019), "Crack segmentation on UAS-based imagery using transfer learning", Proceedings of 2019 International Conference on Image and Vision Computing New Zealand (IVCNZ), pp. 1-6. https://doi.org/10.1109/IVCNZ48456.2019.8960998
  11. Bloice, M.D., Roth, P.M. and Holzinger, A. (2019), "Biomedical image augmentation using Augmentor", Bioinformatics, 35(21), 4522-4524. https://doi.org/10.1093/bioinformatics/btz259
  12. Castro, E., Cardoso, J.S. and Pereira, J.C. (2018), "Elastic deformations for data augmentation in breast cancer mass detection", Proceedings of IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, USA, March, pp. 230-234. https://doi.org/10.1109/BHI.2018.8333411
  13. Cha, Y., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  14. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  15. Chen, F. and Jahanshahi, M.R. (2018), "NB-CNN: deep learning-based crack detection using convolutional neural network and naive Bayes data fusion", IEEE Transact. Indust. Electro., 65(5), 4392-4400. https://doi.org/10.1109/TIE.2017.2764844
  16. Chen, F., Jahanshahi, M.R., Wu, R. and Joffe, C. (2017a), "A texture-based video processing methodology using Bayesian data fusion for autonomous crack detection on metallic surfaces", Comput.-Aided Civil Infrastr. Eng., 32(4), 271-287. https://doi.org/10.1111/mice.12256
  17. Chen, L., Papandreou, G., Schroff, F. and Adam, H. (2017b), "Rethinking atrous convolution for semantic image segmentation", arXiv preprint, http://arxiv.org/abs/1706.05587.
  18. Choi, W. and Cha, Y. (2020), "SDDNet: Real-time crack segmentation", IEEE Transact. Indust. Electro., 67(9), 8016-8025. https://doi.org/10.1109/TIE.2019.2945265
  19. Dong, C.Z. and Catbas, F.N. (2020), "A review of computer vision-based structural health monitoring at local and global levels", Struct. Health Monitor., 20(2), 692-743. https://doi.org/10.1177/1475921720935585
  20. Fan, R., Bocus, M.J., Zhu, Y., Jiao, J., Wang, L., Ma, F., Cheng, S. and Liu, M. (2019), "Road crack detection using deep convolutional neural network and adaptive thresholding", IEEE Intelligent Vehicles Symposium (IV), Paris, France, June, pp. 474-479. https://doi.org/10.1109/IVS.2019.8814000
  21. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z. and Lu, H. (2019), "Dual attention network for scene segmentation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146-3154.
  22. Gerstner, W. and Kistler, W.M. (2002), Spiking Neuron Models: Single Neurons, Populations, Plasticity, Cambridge University Press.
  23. Gidon, A., Zolnik, T.A., Fidzinski, P., Bolduan, F., Papoutsi, A., Poirazi, P., Holtkamp, M., Vida, I. and Larkum, M.E. (2020), "Dendritic action potentials and computation in human layer 2/3 cortical neurons", Science, 367(6473), 83-87. https://doi.org/10.1126/science.aax6239
  24. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, December, pp. 770-778.
  25. Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K.Q. (2017), "Densely connected convolutional networks", Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition CVPR 2017, January, pp. 2261-2269.
  26. Huang, H., Li, Q. and Zhang, D. (2018), "Deep learning based image recognition for crack and leakage defects of metro shield tunnel", Tunnel. Undergr. Space Technol., 77, 166-176. https://doi.org/10.1016/j.tust.2018.04.002
  27. IPC-SHM (2020), http://www.schm.org.cn/#/IPC-SHM,2020/dataDownload
  28. Jahanshahi, M.R. and Masri, S.F. (2013), "A new methodology for non-contact accurate crack width measurement through photogrammetry for automated structural safety evaluation", Smart Mater. Struct., 22(3), 035019. https://doi.org/10.1088/0964-1726/22/3/035019
  29. Jahanshahi, M.R., Kelly, J.S., Masri, S.F. and Sukhatme, G.S. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastr. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930
  30. Jahanshahi, M.R., Masri, S.F., Padgett, C.W. and Sukhatme, G.S. (2013a), "An innovative methodology for detection and quantification of cracks through incorporation of depth perception", Mach. Vision Applicat., 24(2), 227-241. https://doi.org/10.1007/s00138-011-0394-0
  31. Jahanshahi, M.R., Jazizadeh, F., Masri, S.F. and Becerik-Gerber, B. (2013b), "Unsupervised approach for autonomous pavement-defect detection and quantification using an inexpensive depth sensor", J. Comput. Civil Eng., 27(6), 743-754. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000245
  32. Jiang, S. and Zhang, J. (2020), "Real-time crack assessment using deep neural networks with wall-climbing unmanned aerial system", Comput.-Aided Civil Infrastr. Eng., 35(6), 549-564. https://doi.org/10.1111/mice.12519
  33. Kang, D., Benipal, S.S., Gopal, D.L. and Cha, Y.J. (2020), "Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning", Automat. Constr., 118, 103291. https://doi.org/10.1016/j.autcon.2020.103291
  34. Kong, X. and Li, J. (2018), "Vision-based fatigue crack detection of steel structures using video feature tracking", Comput.-Aided Civil Infrastr. Eng., 33(9), 783-799. https://doi.org/10.1111/mice.12353
  35. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012), "ImageNet classification with deep convolutional neural networks", Proceedings of International Conference on Neural Information Processing Systems, pp. 1097-1105.
  36. Li, S., Zhao, X. and Zhou, G. (2019), "Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 34(7), 616-634. https://doi.org/10.1111/mice.12433
  37. Li, G., Ren, X., Qiao, W., Ma, B. and Li, Y. (2020), "Automatic bridge crack identification from concrete surface using resnext with postprocessing", Struct. Control Health Monitor., 27(11), e2620. https://doi.org/10.1002/stc.2620
  38. Lim, R.S., La, H.M. and Sheng, W. (2014), "A robotic crack inspection and mapping system for bridge deck maintenance", IEEE Transactions on Automation Science and Engineering, 11(2), 367-378. https://doi.org/10.1109/TASE.2013.2294687
  39. Lin, F., Yang, J., Shu, J. and Scherer, R.J. (2019), "Crack semantic segmentation using the U-net with full attention strategy", arXiv preprint arXiv:2104.14586.
  40. Long, J., Shelhamer, E. and Darrell, T. (2015), "Fully convolutional networks for semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440.
  41. Makantasis, K., Protopapadakis, E., Doulamis, A., Doulamis, N. and Loupos, C. (2015), "Deep convolutional neural networks for efficient vision based tunnel inspection", Proceedings of 2015 IEEE 11th International Conference on Intelligent Computer Communication and Processing ICCP 2015, Cluj-Napoca, Romania, September, pp. 335-342. https://doi.org/10.1109/ICCP.2015.7312681
  42. Mokhtari, S., Wu, L. and Yun, H.B. (2017), "Statistical selection and interpretation of imagery features for computer vision-based pavement crack-detection systems", J. Perform. Constr. Facil., 31(5), 04017054. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001006
  43. Mukkamala, M.C. and Hein, M. (2017), "Variants of RMSprop and Adagrad with logarithmic regret bounds", Proceedings of International Conference on Machine Learning, pp. 2545-2553.
  44. Ni, F., Zhang, J. and Chen, Z. (2019), "Pixel-level crack delineation in images with convolutional feature fusion", Struct. Control Health Monitor., 26(1), 1-18. https://doi.org/10.1002/stc.2286
  45. Nishikawa, T., Yoshida, J., Sugiyama, T. and Fujino, Y. (2012), "Concrete crack detection by multiple sequential image filtering", Comput.-Aided Civil Infrastr. Eng., 27(1), 29-47. https://doi.org/10.1111/j.1467-8667.2011.00716.x
  46. Pan, Y., Zhang, G. and Zhang, L. (2020), "A spatial-channel hierarchical deep learning network for pixel-level automated crack detection", Automat. Constr., 119, 103357. https://doi.org/10.1016/j.autcon.2020.103357
  47. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-net: convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
  48. Shi, Y., Cui, L., Qi, Z., Meng, F. and Chen, Z. (2016), "Automatic road crack detection using random structured forests", IEEE Transactions on Intelligent Transportation Systems, 17(12), 3434-3445. https://doi.org/10.1109/TITS.2016.2552248
  49. Simard, P.Y., Steinkraus, D. and Platt, J.C. (2003), "Best practices for convolutional neural networks applied to visual document analysis", Proceedings of the International Conference on Document Analysis and Recognition ICDAR, January, pp. 958-963.
  50. Simonyan, K. and Zisserman, A. (2014), "Very deep convolutional networks for large-scale image recognition", arXiv preprint, arXiv: 1409.1556.
  51. Soukup, D. and Huber-Mork, R. (2014), "Convolutional neural networks for steel surface defect detection from photometric stereo images", (Bebis G. et al. eds.), In: Advances in Visual Computing, pp. 668-677.
  52. Spencer, B.F. Jr, Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  53. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. (2015), "Going deeper with convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9.
  54. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016), "Rethinking the inception architecture for computer vision", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, December, pp. 2818-2826.
  55. VanRullen, R., Guyonneau, R. and Thorpe, S.J. (2005), "Spike times make sense", Trends Neurosci., 28(1), 1-4. https://doi.org/10.1016/j.tins.2004.10.010
  56. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", In: Advances in Neural Information Processing Systems, pp. 5998-6008.
  57. Wan, H., Gao, L., Su, M., Sun, Q. and Huang, L. (2021), "Attention-based convolutional neural network for pavement crack detection", Adv. Mater. Sci. Eng. https://doi.org/10.1155/2021/5520515
  58. Wang, Z. and Cha, Y. (2020), "Unsupervised deep learning approach using a deep auto-encoder with an one-class support vector machine to detect structural damage", Struct. Health Monitor., 20(1), 406-425. https://doi.org/10.1177/1475921720934051
  59. Wang, D., Dong, Y., Pan, Y. and Ma, R. (2020), "Machine vision-based monitoring methodology for the fatigue cracks in U-ribto-deck weld seams", IEEE Access, 8, 94204-94219. https://doi.org/10.1109/ACCESS.2020.2995276
  60. Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2018), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monitor., 25(2), e2075. https://doi.org/10.1002/stc.2075
  61. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2019a), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images", Struct. Health Monitor., 18(3), 653-674. https://doi.org/10.1177/1475921718764873
  62. Xu, Y., Wei, S., Bao, Y. and Li, H. (2019b), "Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network", Struct. Control Health Monitor., 26(3), e2313. https://doi.org/10.1002/stc.2313
  63. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  64. Yeum, C.M. and Dyke, S.J. (2015), "Vision-based automated crack detection for bridge inspection", Comput.-Aided Civil Infrastr. Eng., 30(10), 759-770. https://doi.org/10.1111/mice.12141
  65. Yeum, C.M., Dyke, S.J. and Ramirez, J. (2018), "Visual data classification in post-event building reconnaissance", Eng. Struct., 155, 16-24. https://doi.org/10.1016/j.engstruct.2017.10.057
  66. Zhang, S.Q. and Zhou, Z.H. (2020), "Flexible transmitter network", arXiv preprint, arXiv:2004.03839.
  67. Zhang, L., Yang, F., Daniel Zhang, Y. and Zhu, Y.J. (2016), "Road crack detection using deep convolutional neural network", Proceedings of International Conference on Image Processing ICIP, August, pp. 3708-3712. https://doi.org/10.1109/ICIP.2016.7533052
  68. Zhang, X., Rajan, D. and Story, B. (2019), "Concrete crack detection using context-aware deep semantic segmentation network", Comput.-Aided Civil Infrastr. Eng., 34(11), 951-971. https://doi.org/10.1111/mice.12477