Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

An active learning method with difficulty learning mechanism for crack detection

  • Shu, Jiangpeng (College of Civil Engineering and Architecture, Zhejiang University) ;
  • Li, Jun (College of Civil Engineering and Architecture, Zhejiang University) ;
  • Zhang, Jiawei (College of Civil Engineering and Architecture, Zhejiang University) ;
  • Zhao, Weijian (College of Civil Engineering and Architecture, Zhejiang University) ;
  • Duan, Yuanfeng (College of Civil Engineering and Architecture, Zhejiang University) ;
  • Zhang, Zhicheng (College of Civil Engineering and Architecture, Zhejiang University)
  • Received : 2021.04.28
  • Accepted : 2021.08.03
  • Published : 2022.01.25
⟨ Previous Next ⟩

Abstract

Crack detection is essential for inspection of existing structures and crack segmentation based on deep learning is a significant solution. However, datasets are usually one of the key issues. When building a new dataset for deep learning, laborious and time-consuming annotation of a large number of crack images is an obstacle. The aim of this study is to develop an approach that can automatically select a small portion of the most informative crack images from a large pool in order to annotate them, not to label all crack images. An active learning method with difficulty learning mechanism for crack segmentation tasks is proposed. Experiments are carried out on a crack image dataset of a steel box girder, which contains 500 images of 320×320 size for training, 100 for validation, and 190 for testing. In active learning experiments, the 500 images for training are acted as unlabeled image. The acquisition function in our method is compared with traditional acquisition functions, i.e., Query-By-Committee (QBC), Entropy, and Core-set. Further, comparisons are made on four common segmentation networks: U-Net, DeepLabV3, Feature Pyramid Network (FPN), and PSPNet. The results show that when training occurs with 200 (40%) of the most informative crack images that are selected by our method, the four segmentation networks can achieve 92%-95% of the obtained performance when training takes place with 500 (100%) crack images. The acquisition function in our method shows more accurate measurements of informativeness for unlabeled crack images compared to the four traditional acquisition functions at most active learning stages. Our method can select the most informative images for annotation from many unlabeled crack images automatically and accurately. Additionally, the dataset built after selecting 40% of all crack images can support crack segmentation networks that perform more than 92% when all the images are used.

Keywords

Acknowledgement

The authors would like to thank the organizations of the International Project Competition for SHM (IPC-SHM 2020) ANCRiSST, Harbin Institute of Technology (China), and University of Illinois at Urbana-Champaign (USA) for their generously providing the invaluable data from actual structures. The authors would also like to gratefully acknowledge the support from the National Key R&D Program of China (2018YFE0125400) and the National Natural Science Foundation of China (U1709216), which made the research possible.

References

  1. Azimi, M., Eslamlou, A.D. and Pekcan, G. (2020), "Data-driven structural health monitoring and damage detection through deep learning: state-of-the-art review", Sensors, 20(10), 2778. https://doi.org/10.3390/s20102778
  2. Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z. and Li, H. (2019), "The state of the art of data science and engineering in structural health monitoring", Engineering, 5(2), 234-242. https://doi.org/10.1016/j.eng.2018.11.027
  3. Bao, Y., Li, J., Nagayama, T., Xu, Y., Spencer Jr., B.F. and Li, H. (2021), "The 1st International Project Competition for Structural Health Monitoring (IPC-SHM, 2020): A summary and benchmark problem", Struct. Health Monitor., 20(4), 2229-2239. https://doi.org/10.1016/j.eng.2018.11.027
  4. Cai, W. and Wei, Z. (2020), "Remote sensing image classification based on a cross-attention mechanism and graph convolution", IEEE Geosci. Remote Sens. Lett. https://doi.org/10.1109/LGRS.2020.3026587
  5. Chaudhari, S., Polatkan, G., Ramanath, R. and Mithal, V. (2019), "An attentive survey of attention models", arXiv preprint arXiv: 1904.02874. https://arxiv.org/abs/1904.02874
  6. Chen, L.-C., Papandreou, G., Schroff, F. and Adam, H. (2017), "Rethinking atrous convolution for semantic image segmentation", arXiv preprint arXiv: 1706.05587. https://arxiv.org/abs/1706.05587
  7. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018), "Encoder-decoder with atrous separable convolution for semantic image segmentation", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September.
  8. Fan, G., Li, J. and Hao, H. (2019), "Lost data recovery for structural health monitoring based on convolutional neural networks", Struct. Control Health Monitor., 26(10), 1-21. https://doi.org/10.1002/stc.2433
  9. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z. and Lu, H. (2019), "Dual attention network for scene segmentation", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June.
  10. Fujita, Y., Mitani, Y. and Hamamoto, Y. (2006), "A method for crack detection on a concrete structure", Proceedings of International Conference on Pattern Recognition, Hong Kong, China, August.
  11. Hadidi, N.N., Cullen, K.R., Hall, L.M.J., Lindquist, R., Buckwalter, K.C. and Mathews, E. (2014), "Functional magnetic resonance imaging as experienced by stroke survivors", Res. Gerontol. Nurs., 7(5), 200-205. https://doi.org/10.3928/19404921-20140820-01
  12. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June.
  13. Kang, D. and Cha, Y.J. (2018), "Autonomous uavs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging", Comput.-Aided Civil Infrastr. Eng., 33(10), 885-902. https://doi.org/10.1111/mice.12375
  14. Kim, H., Ahn, E., Shin, M. and Sim, S.H. (2019), "Crack and noncrack classification from concrete surface images using machine learning", Struct. Health Monitor., 18(3), 725-738. https://doi.org/10.1177/1475921718768747
  15. Kuo, W., Hane, C., Yuh, E., Mukherjee, P. and Malik, J. (2018), "Cost-sensitive active learning for intracranial hemorrhage detection", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, September.
  16. Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B. and Belongie, S. (2017), "Feature pyramid networks for object detection", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.
  17. Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019), "Computer vision-based concrete crack detection using U-net fully convolutional networks", Automat. Constr., 104, 129-139. https://doi.org/10.1016/j.autcon.2019.04.005
  18. Mackowiak, R., Lenz, P., Ghori, O., Diego, F., Lange, O. and Rother, C. (2018), "Cereals-cost-effective region-based active learning for semantic segmentation", arXiv preprint arXiv: 1810.09726. https://arxiv.org/abs/1810.09726
  19. Milletari, F., Navab, N. and Ahmadi, S.A. (2016), "V-Net: Fully convolutional neural networks for volumetric medical image segmentation", Proceedings of International Conference on 3D Vision, Stanford, CA, USA, October.
  20. Modarres, C., Astorga, N., Droguett, E.L. and Meruane, V. (2018), "Convolutional neural networks for automated damage recognition and damage type identification", Struct. Control Health Monitor., 25(10), 1-17. https://doi.org/10.1002/stc.2230
  21. Nguyen, H.N., Kam, T.Y. and Cheng, P.Y. (2014), "An automatic approach for accurate edge detection of concrete crack utilizing 2d geometric features of crack", J. Signal Process. Syst., 77(3), 221-240. https://doi.org/10.1007/s11265-013-0813-8
  22. Oliveira, H. and Correia, P.L. (2009), "Automatic road crack segmentation using entropy and image dynamic thresholding", Proceedings of European Signal Processing Conference, Glasgow, Scotland, UK, August.
  23. Pan, Y., Zhang, G. and Zhang, L. (2020), "A spatial-channel hierarchical deep learning network for pixel-level automated crack detection", Automat. Constr., 119, 103357. https://doi.org/10.1016/j.autcon.2020.103357
  24. Pathirage, C.S.N., Li, J., Li, L., Hao, H., Liu, W. and Wang, R. (2019), "Development and application of a deep learning-based sparse autoencoder framework for structural damage identification", Struct. Health Monitor., 18(1), 103-122. https://doi.org/10.1177/1475921718800363
  25. Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Chen, X. and Wang, X. (2020), "A survey of deep active learning", arXiv preprint arXiv: 2009.00236. https://arxiv.org/abs/2009.00236
  26. Ronneberger, O., Fischer, P. and Brox, T. (2015), "U-Net: convolutional networks for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October.
  27. Sener, O. and Savarese, S. (2017), "Active learning for convolutional neural networks: A core-set approach", arXiv preprint arXiv: 1708.00489. https://arxiv.org/abs/1708.00489
  28. Siddiqui, Y., Valentin, J. and Niessner, M. (2020), "Viewal: Active learning with viewpoint entropy for semantic segmentation", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, USA, June.
  29. Sinha, S., Ebrahimi, S. and Darrell, T. (2019), "Variational adversarial active learning", Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, November.
  30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", Proceedings of Advances in Neural Information Processing Systems, Long Beach, CA, USA, December.
  31. Wang, K., Zhang, D., Li, Y., Zhang, R. and Lin, L. (2017), "Cost-Effective active learning for deep image classification", IEEE Transact. Circuits Syst. Video Technol., 27(12), 2591-2600 https://doi.org/10.1109/TCSVT.2016.2589879
  32. Wang, X., Girshick, R., Gupta, A. and He, K. (2018), "Non-local neural networks", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June.
  33. Wang, Z., Xu, G., Ding, Y., Wu, B. and Lu, G. (2020), "A vision-based active learning convolutional neural network model for concrete surface crack detection", Adv. Struct. Eng., 23(13), 2952-2964. https://doi.org/10.1177/1369433220924792
  34. Xie, S., Girshick, R., Dollar, P., Tu, Z. and He, K. (2017), "Aggregated residual transformations for deep neural networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.
  35. Xie, S., Feng, Z., Chen, Y., Sun, S., Ma, C. and Song, M. (2020), "DEAL: Difficulty-aware Active Learning for Semantic Segmentation", Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan, December.
  36. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2019), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images", Struct. Health Monitor., 18(3), 653-674. https://doi.org/10.1177/1475921718764873
  37. Yang, L., Zhang, Y., Chen, J., Zhang, S. and Chen, D.Z. (2017), "Suggestive annotation: A deep active learning framework for biomedical image segmentation", Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, Toronto, Canada, September.
  38. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  39. Ye, X.W., Jin, T. and Yun, C.B. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst., Int. J., 24(5), 567-586. https://doi.org/10.12989/sss.2019.24.5.567
  40. Yoo, D. and Kweon, I.S. (2019), "Learning loss for active learning", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June.
  41. Zhang, Y., Gao, J. and Zhou, H. (2012), "ImageNet classification with deep convolutional neural networks", Adv. Neural Inform. Process. Syst., 25, 1097-1105.
  42. Zhang, X., Rajan, D. and Story, B. (2019), "Concrete crack detection using context-aware deep semantic segmentation network", Comput.-Aided Civil Infrastr. Eng., 34(11), 951-971. https://doi.org/10.1111/mice.12477
  43. Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J. (2017), "Pyramid scene parsing network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.
  44. Zhou, C., Bai, J., Song, J., Liu, X., Zhao, Z., Chen, X. and Gao, J. (2018), "ATRank: An attention-based user behavior modeling framework for recommendation", Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, February.