Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A deep and multiscale network for pavement crack detection based on function-specific modules

  • Guolong Wang (Department of Civil and Environmental Engineering, Oklahoma State University) ;
  • Kelvin C.P. Wang (Department of Civil and Environmental Engineering, Oklahoma State University) ;
  • Allen A. Zhang (Department of Civil Engineering, Southwest Jiaotong University) ;
  • Guangwei Yang (Department of Civil and Environmental Engineering, Oklahoma State University)
  • Received : 2021.12.07
  • Accepted : 2023.08.14
  • Published : 2023.09.25
⟨ Previous Next ⟩

Abstract

Using 3D asphalt pavement surface data, a deep and multiscale network named CrackNet-M is proposed in this paper for pixel-level crack detection for improvements in both accuracy and robustness. The CrackNet-M consists of four function-specific architectural modules: a central branch net (CBN), a crack map enhancement (CME) module, three pooling feature pyramids (PFP), and an output layer. The CBN maintains crack boundaries using no pooling reductions throughout all convolutional layers. The CME applies a pooling layer to enhance potential thin cracks for better continuity, consuming no data loss and attenuation when working jointly with CBN. The PFP modules implement direct down-sampling and pyramidal up-sampling with multiscale contexts specifically for the detection of thick cracks and exclusion of non-crack patterns. Finally, the output layer is optimized with a skip layer supervision technique proposed to further improve the network performance. Compared with traditional supervisions, the skip layer supervision brings about not only significant performance gains with respect to both accuracy and robustness but a faster convergence rate. CrackNet-M was trained on a total of 2,500 pixel-wise annotated 3D pavement images and finely scaled with another 200 images with full considerations on accuracy and efficiency. CrackNet-M can potentially achieve crack detection in real-time with a processing speed of 40 ms/image. The experimental results on 500 testing images demonstrate that CrackNet-M can effectively detect both thick and thin cracks from various pavement surfaces with a high level of Precision (94.28%), Recall (93.89%), and F-measure (94.04%). In addition, the proposed CrackNet-M compares favorably to other well-developed networks with respect to the detection of thin cracks as well as the removal of shoulder drop-offs.

Keywords

Acknowledgement

The study presented in this article was partially supported by the National Natural Science Foundation of China (Grant No. 51208419) and the Fundamental Research Funds for Central Universities of China (Grant No. 2682021CX009).

References

  1. Alipour, M., Harris, D.K. and Miller, G.R. (2019), "Robust pixel-level crack detection using deep fully convolutional neural networks", J. Comput. Civil Eng., 33(6), 04019040. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000854
  2. Asadi, P., Mehrabi, H., Asadi, A. and Ahmadi, M. (2021), "Deep convolutional neural networks for pavement crack detection using an inexpensive global shutter RGB-D sensor and ARM-based single-board computer", Transport. Res. Record, 2675(9) 03611981211004974. https://doi.org/10.1177/03611981211004974
  3. Badrinarayanan, V., Handa, A. and Cipolla, R. (2015), "SegNet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling", arXiv preprint arXiv:1505.07293. https://doi.org/10.48550/arXiv.1505.07293
  4. Bang, S., Park, S., Kim, H. and Kim, H. (2019), "Encoder-decoder network for pixel-level road crack detection in blackbox images", Comput.-Aided Civil Infrastr. Eng., 34(8), 713-727. https://doi.org/10.1111/mice.12440
  5. Carre, A., Klausner, G., Edjlali, M., Lerousseau, M., Briend-Diop, J., Sun, R., Ammari, S., Reuze, S., Alvarez Andres, E., Estienne, T. and Niyoteka, S. (2020), "Standardization of brain MR images across machines and protocols: bridging the gap for MRI-based radiomics", Scientific Reports, 10(1), 12340.
  6. Cha, Y.J., Choi, W. and Buyukozturk, O. (2017), "Deep learning-based crack damage detection using convolutional neural networks", Comput.-Aided Civil Infrastr. Eng., 32(5), 361-378. https://doi.org/10.1111/mice.12263
  7. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput.-Aided Civil Infrastr. Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  8. Daniel, A. and Preeja, V. (2014), "A novel technique for automatic road distress detection and analysis", Int. J. Comput. Applicat., 101(10), 18-23. https://doi.org/10.5120/17723-8018
  9. Doshi, K. and Yasin, Y. (2020), "Road damage detection using deep ensemble learning", In: 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA, December.
  10. Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y. and Kang, H. (2020), "Pavement distress detection and classification based on YOLO network", Int. J. Pave. Eng., 22(13), 1659-1672. https://doi.org/10.1080/10298436.2020.1714047
  11. Escalona, U., Arce, F., Zamora, E. and Sossa, H. (2019), "Fully convolutional networks for automatic pavement crack segmentation", Computacion y Sistemas, 23(2), 451-460. https://doi.org/10.13053/cys-23-2-3047
  12. Fawcett, T. (2006), "An introduction to ROC analysis", Pattern Recogn. Lett., 27(8), 861-874. https://doi.org/10.1016/j.patrec.2005.10.010
  13. Fei, Y., Wang, K.C.P., Zhang, A., Chen, C., Li, J.Q., Liu, Y., Yang, G. and Li, B. (2019), "Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V", IEEE Transact. Intell. Transport. Syst., 21(1), 273-284. https://ieeexplore.ieee.org/document/8620557
  14. Feng, X., Xiao, L., Li, W., Pei, L., Sun, Z., Ma, Z., Shen, H. and Ju, H. (2020), "Pavement crack detection and segmentation method based on improved deep learning fusion model", Mathe. Problems Eng., 2020. https://doi.org/10.1155/2020/8515213
  15. Gao, Y., Zhai, P. and Mosalam, K.M. (2021), "Balanced semisupervised generative adversarial network for damage assessment from low-data imbalanced-class regime", Comput.-Aided Civil Infrastr. Eng., 36(9), 1094-1113. https://doi.org/10.1111/mice.12741
  16. Glorot, X. and Bengio, Y. (2010), "Understanding the difficulty of training deep feedforward neural networks", Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, May.
  17. Gou, C., Peng, B., Li, T. and Gao, Z. (2019), "Pavement Crack Detection Based on the Improved Faster-RCNN", Proceedings of 2019 IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering (ISKE), Dalian, China, November.
  18. Guo, C., Fan, B., Zhang, Q., Xiang, S. and Pan, C. (2019), "AugFPN: improving multi-scale feature learning for object detection", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CA, USA, June.
  19. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June.
  20. Hsieh, Y.A. and Tsai, Y.J. (2020), "Machine learning for crack detection: review and model performance comparison", J. Comput. Civil Eng., 34(5), 04020038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918
  21. Hwang, J.J. and Liu, T.L. (2015), "Pixel-wise deep learning for contour detection", arXiv preprint arXiv:1504.01989. https://doi.org/10.48550/arXiv.1504.01989
  22. Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J. and Keutzer, K. (2016), "Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 MB model size", arXiv preprint arXiv:1602.07360. https://doi.org/10.48550/arXiv.1602.07360
  23. Jahanshahi, M.R., Jazizadeh, F., Masri, S.F. and Becerik-Gerber, B. (2013), "Unsupervised approach for autonomous pavement-defect detection and quantification using an inexpensive depth sensor", J. Comput. Civil Eng., 27(6), 743-754. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000245
  24. Kingma, D.P. and Ba, J.L. (2015), "Adam: a method for stochastic optimization", arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980
  25. Konig, J., Jenkins, M.D., Barrie, P., Mannion, M. and Morison, G. (2019), "A convolutional neural network for pavement surface crack segmentation using residual connections and attention gating", In: 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, September.
  26. Lee, B.J. and Lee, H.D. (2004), "Position-invariant neural network for digital pavement crack analysis", Comput.-Aided Civil Infrastr. Eng., 19(2), 105-118. https://doi.org/10.1111/j.1467-8667.2004.00341.x
  27. Li, B., Wang, K.C.P., Zhang, A., Yue, F. and Giuseppe, S. (2019a), "Automatic segmentation and enhancement of pavement cracks based on 3D pavement images", J. Adv. Transport., 2019, 1813763. https://doi.org/10.1155/2019/1813763
  28. Li, S., Zhao, X. and Zhou, G. (2019b), "Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 34, 616-634. https://doi.org/10.1111/mice.12433
  29. Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B. and Belongie, S. (2016), "Feature pyramid networks for object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, AZ, USA, June.
  30. Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2017), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, October.
  31. Ma, K., Hoai, M. and Samaras, D. (2017), "Large-scale continual road inspection: visual infrastructure assessment in the wild", In: BMVC. https://dx.doi.org/10.5244/C.31.151
  32. Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018), "Road damage detection using deep neural networks with images captured through a smartphone", arXiv preprint arXiv:1801.09454. https://doi.org/10.48550/arXiv.1801.09454
  33. Mohammed Y., Uddin, N., Tan, C. and Shi, Z. (2020), "Crack detection using faster R-CNN and point feature matching", Civil Eng. Res. J., 10(3), 555790.
  34. Munawar, H.S., Hammad, A.W., Haddad, A., Soares, C.A.P. and Waller, S.T. (2021), "Image-based crack detection methods: A review", Infrastructures, 6(8), 115. https://doi.org/10.3390/infrastructures6080115
  35. Sathya, K., Sangavi, D., Sridharshini, P., Manobharathi, M. and Jayapriya, G. (2020), "Improved image based super resolution and concrete crack prediction using pre-trained deep learning models", J. Soft Comput. Civil Eng., 4(3), 34-44. https://doi.org/10.22115/SCCE.2020.229355.1219
  36. Shi, Y., Cui, L., Qi, Z., Meng, F. and Chen, Z. (2016), "Automatic road crack detection using random structured forests", IEEE Transact. Intell. Transport. Syst., 17(12), 3434-3445. https://ieeexplore.ieee.org/document/7471507 https://doi.org/10.1109/TITS.2016.2552248
  37. Sizyakin, R., Voronin, V., Gapon, N. and Pizurica, A. (2020), "A deep learning approach to crack detection on road surfaces", In: Artificial Intelligence and Machine Learning in Defense Applications II, 11543(115430P), 1-7. https://doi.org/10.1117/12.2574131
  38. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. (2014), "Going deeper with convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, June.
  39. Tan, M. and Le, Q. (2019), "Efficientnet: Rethinking model scaling for convolutional neural networks", In: International Conference on Machine Learning, CA, USA, June.
  40. Wang, G., Wang, K.C.P., Yang, G., Liu, Y., Li, J.Q. and Peters, W. (2022), "Nondestructive bridge deck evaluation using sub-mm 3D laser imaging technology at highway speeds", J. Bridge Eng., 27(6), 04022045. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001888
  41. Xu, H., Su, X., Wang, Y., Cai, H., Cui, K. and Chen, X. (2019), "Automatic bridge crack detection using a convolutional neural network", Appl. Sci., 9(14), 2867. https://doi.org/10.3390/app9142867
  42. Yang, X., Li, H., Yu, Y., Luo, X. and Huang, T. (2018), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput.-Aided Civil Infrastr. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  43. Yang, F., Zhang, L., Yu, S., Prokhorov, D., Mei, X. and Ling, H. (2019), "Feature pyramid and hierarchical boosting network for pavement crack detection", IEEE Transact. Intell. Transport. Syst., 21(4), 1525-1535. https://doi.org/10.48550/arXiv.1901.06340
  44. Ye, X.W., Jin, T. and Yun, C.B. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst., Int. J., 24(5), 567-586. https://doi.org/10.12989/sss.2019.24.5.567
  45. Zalama, E., Gomez-Garcia-Bermejo, J., Medina, R. and Llamas, J. (2014), "Road crack detection using visual features extracted by Gabor filters", Comput.-Aided Civil Infrastr. Eng., 29(5), 342-358. https://doi.org/10.1111/mice.12042
  46. Zhang, A., Wang, K.C.P., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J Q. and Chen, C. (2017), "Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network", Comput.-Aided Civil Infrastr. Eng., 32(10), 805-819. https://doi.org/10.1111/mice.12297
  47. Zhang, A., Wang, K. C. P., Fei, Y., Liu, Y., Tao, S., Chen, C., Li, J. Q. and Li, B. (2018), "Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet", J. Comput. Civil Eng., 32(5), 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775
  48. Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J. (2017), "Pyramid scene parsing network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.