Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A vision-based system for inspection of expansion joints in concrete pavement

  • Jung Hee Lee (Department of Artificial Intelligence, Ajou University) ;
  • bragimov Eldor (Research and Development Division, SISTech Co. Ltd) ;
  • Heungbae Gil (ICT Convergence Research Division, Korea Expressway Corporation Research Institute) ;
  • Jong-Jae Lee (Department of Civil and Environmental Engineering, Sejong University)
  • Received : 2023.06.30
  • Accepted : 2023.10.24
  • Published : 2023.11.25
⟨ Previous Next ⟩

Abstract

The appropriate maintenance of highway roads is critical for the safe operation of road networks and conserves maintenance costs. Multiple methods have been developed to investigate the surface of roads for various types of cracks and potholes, among other damage. Like road surface damage, the condition of expansion joints in concrete pavement is important to avoid unexpected hazardous situations. Thus, in this study, a new system is proposed for autonomous expansion joint monitoring using a vision-based system. The system consists of the following three key parts: (1) a camera-mounted vehicle, (2) indication marks on the expansion joints, and (3) a deep learning-based automatic evaluation algorithm. With paired marks indicating the expansion joints in a concrete pavement, they can be automatically detected. An inspection vehicle is equipped with an action camera that acquires images of the expansion joints in the road. You Only Look Once (YOLO) automatically detects the expansion joints with indication marks, which has a performance accuracy of 95%. The width of the detected expansion joint is calculated using an image processing algorithm. Based on the calculated width, the expansion joint is classified into the following two types: normal and dangerous. The obtained results demonstrate that the proposed system is very efficient in terms of speed and accuracy.

Keywords

Acknowledgement

The authors acknowledge the editorial service of Mr. Ryan Lee at Cate School, Carpinteria, CA, USA such as translation, rephrasing, and Fig. drawings.

References

  1. Abu-Ain, W., Abdullah, S.N.H.S., Bataineh, B., Abu-Ain, T. and Omar, K. (2013), "Skeletonization algorithm for binary images", Procedia Technol., 11, 704-709. https://doi.org/10.1016/j.protcy.2013.12.248
  2. Arya, D., Maeda, H., Ghosh, S.K., Toshniwal, D., Mraz, A., Kashiyama, T. and Sekimoto, Y. (2021), "Deep learning-based road damage detection and classification for multiple countries", Autom. Constr., 132, 103935. https://doi.org/10.1016/j.autcon.2021.103935
  3. Dai, J., Li, Y., He, K. and Sun, J. (2016), "R-fcn: Object detection via region-based fully convolutional networks", Proceedings of the Neural Information Processing Systems Conference (NIPS), pp. 379-387.
  4. Dung, C.V. and Anh, L.D. (2019), "Autonomous concrete crack detection using deep fully convolutional neural network", Autom. Constr., 99, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  5. Durix, B., Chambon, S., Leonard, K., Mari, J.L. and Morin, G. (2019), "The propagated skeleton: A robust detail-preserving approach", Proceedings of Discrete Geometry for Computer Imagery: 21st IAPR International Conference, DGCI 2019, Marne-la-Vallee, France, March, pp. 343-354. https://doi.org/10.1007/978-3-030-14085-4_27
  6. Gao, S., Jie, Z., Pan, Z., Qin, F. and Li, R. (2018), "Automatic recognition of pavement crack via convolutional neural network", In: Lecture Notes Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 82-89. https://doi.org/10.1007/978-3-662-56689-3_7
  7. Guo, Z. and Hall, R.W. (1989), "Parallel thinning with two-subiteration algorithms", Commun. ACM, 32(3), 359-373. https://doi.org/10.1145/62065.62074
  8. Handalage, U. and Kuganandamurthy, L. (2021), "Real-time object detection using YOLO: A review", ResearchGate.
  9. Haralick, R.M., Sternberg, S.R. and Zhuang, X. (1987), "Image analysis using mathematical morphology", IEEE Transact. Pattern Anal. Mach. Intell., PAMI-9(4), 532-550. https://doi.org/10.1109/TPAMI.1987.4767941
  10. He, K., Zhang, X., Ren, S. and Sun, J. (2015), "Spatial pyramid pooling in deep convolutional networks for visual recognition", IEEE Trans. Pattern Anal. Mach. Intell., 37, 1904-1916. https://doi.org/10.1109/TPAMI.2015.2389824
  11. He, K., Gkioxari, G., Dollar, P. and Girshick, R., (2017), "Mask R-CNN", Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969.
  12. Jiang, P., Ergu, D., Liu, F., Cai, Y. and Ma, B. (2022), "A review of YOLO algorithm developments", Procedia Comput. Sci., 199, 1066-1073. https://doi.org/10.1016/j.procs.2022.01.135
  13. Jocher, G. (2022) YOLOv5 release v6.1, https://github.com/ultralytics/yolov5/releases/tag/v6.0, 2022
  14. Jocher, G., Chaurasia, A. and Qiu, J. (2023), YOLO by Ultralytics (Version 8.0.0). https://github.com/ultralytics/ultralytics
  15. Karballaeezadeh, N., Zaremotekhases, F., Shamshirband, S., Mosavi, A., Nabipour, N., Csiba, P. and Varkonyi-Koczy, A.R. (2020), "Intelligent Road inspection with advanced machine learning; Hybrid prediction models for smart mobility and transportation maintenance systems", Energies, 13. https://doi.org/10.3390/en13071718
  16. Kim, J., Choi, S., Do, M. and Han, D. (2016), "Road maintenance planning with traffic demand forecasting", Int. J. Highw. Eng., 18, 47-57. https://doi.org/10.7855/IJHE.2016.18.3.047
  17. Kim, H., Lee, J., Ahn, E., Cho, S., Shin, M. and Sim, S.H. (2017), "Concrete crack identification using a UAV incorporating hybrid image processing", Sensors, 17(9), 1424-8220. https://doi.org/10.3390/s17092052
  18. Leborgne, A., Mille, J. and Tougne, L. (2015), "Noise-resistant digital euclidean connected skeleton for graph-based shape matchingg", J. Visual Commun. Image Represent, 31, 165-176. https://doi.org/10.1016/j.jvcir.2015.06.005
  19. Lee, S.H. and Yoo, I.K. (2008), "Overview of pavement management system in Korea and its operation results", In: ISARC 2008. Proceedings of the from the 25th International Symposium on Automation and Robotics in Construction, pp. 355-364. https://doi.org/10.3846/isarc.20080626.355
  20. Lee, S.T., Park, D.W. and Mission, J.L. (2013), "Estimation of pavement rehabilitation cost using pavement management data", Struct. Infrastruct. Eng., 9, 458-464. https://doi.org/10.1080/15732479.2011.560163
  21. Li, H., Song, D., Liu, Y. and Li, B. (2019), "Automatic pavement crack detection by multi-scale image fusion", IEEE Trans. Intell. Transp. Syst., 20(6), 2025-2036. https://doi.org/10.1109/TITS.2018.2856928
  22. Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2018), "Focal loss for dense object detection", Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988.
  23. Liu, Y., Yao, J., Lu, X., Xie, R. and Li, L. (2019), "DeepCrack: A deep hierarchical feature learning architecture for crack segmentation", Neurocomputing, 338, 139-153. https://doi.org/10.1016/j.neucom.2019.01.036
  24. Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018), "Road damage detection and classification using deep neural networks with smartphone images", Comput. Aid. Civil Infrastruct. Eng., 33, 1127-1141. https://doi.org/10.1111/mice.12387
  25. Maeda, H., Kashiyama, T., Sekimoto, Y., Seto, T. and Omata, H. (2021), "Generative adversarial network for road damage detection", Comput. Aid. Civil Infrastruct. Eng., 36, 47-60. https://doi.org/10.1111/mice.12561
  26. Nisanth, A. and Mathew, A. (2014), "Automated visual inspection on pavement crack detection and characterization", Int. J. Technol. Eng. Syst., 6(1), 14-20.
  27. Ogniewicz, R.L. and Ilg, M. (1992), "Voronoi skeletons: theory and applications", Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June, pp. 63-69. https://doi.org/10.1109/CVPR.1992.223226
  28. Ong, J.C., Ismadi, M.Z.P. and Wang, X. (2022), "A hybrid method for pavement crack width measurement", Measurement, 197, 111260. https://doi.org/10.1016/j.measurement.2022.111260
  29. Redmon, J. and Farhadi, A. (N.D.) (2022), YOLOv3: An incremental improvement. Available online: https://pjreddie.com/YOLO/ (accessed on Nov 24, 2022)
  30. Ren, S., He, K., Girshick, R. and Sun, J. (2017), "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Trans. Pattern Anal. Mach. Intell., 39, 1137-1149. https://doi.org/10.1109/TPAMI.2016.2577031
  31. Rudzitis, A. and Zaeva, M.A. (2021), "Alternative inverse perspective mapping homography matrix computation for ADAS systems using camera intrinsic and extrinsic calibration parameters", Procedia Comput. Sci., 190, 695-700. https://doi.org/10.1016/J.PROCS.2021.06.081
  32. Shon, H. and Lee, J. (2021), "Integrating multi-scale inspection, maintenance, rehabilitation, and reconstruction decisions into system-level pavement management systems", Transport. Res. Part C: Emerg. Technol., 131, p. 103228. https://doi.org/10.1016/j.trc.2021.103328
  33. Siriborvornratanakul, T. (2018), "An automatic road distress visual inspection system using an onboard in-car camera", Adv. Multimedia, 1-10. https://doi.org/10.1155/2018/2561953
  34. Tayo, C.O., Linsangan, N.B. and Pellegrino, R.V. (2019), "Portable crack width calculation of concrete road pavement using machine vision", Proceedings of the 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Laoag, Philippines, pp. 1-5. https://doi.org/10.1109/HNICEM48295.2019.9072731
  35. Tzutalin. LabelImg. Git code (2015), https://github.com/tzutalin/labelImg
  36. Wang, C.-Y., Bochkovskiy, A. and Liao, H.-Y. M. (2021), "Scaled-yolov4: Scaling cross stage partial network", Proceedings of the IEEE/cvf Conference on Computer Vision and Pattern Recognition, pp. 13029-13038. https://doi.org/10.1109/CVPR46437.2021.01283
  37. Wang, C.Y., Bochkovskiy, A. and Liao, H.Y.M. (2022), "Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7464-7475.
  38. Wu, R.T., Tang, W. and Jahanshahi, M.R. (2022), "Crack segmentation in high-resolution images using cascaded deep convolutional neural networks and Bayesian data fusion", Smart Struct. Syst., Int. J., 29(1), 221-235. https://doi.org/10.12989/sss.2022.29.1.221
  39. Zhang, L., Yang, F., Zhang, Y.D. and Zhu, Y.J. (2016), "Road crack detection using deep convolutional neural network", IEEE International Conference on Image Processing (ICIP), pp. 3708-3712. https://doi.org/10.1109/ICIP.2016.7533052
  40. Zhang, A., Wang, K.C., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J.Q. and Chen, C. (2017), "Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network", Comput.-Aided Civ. Infrastruct. Eng., 32(10), 805-819. https://doi.org/10.1111/mice.12297
  41. Zhao, J., Hu, F., Qiao, W., Zhai, W., Xu, Y., Bao, Y. and Li, H. (2022), "A modified u-net for crack segmentation by self-attention-self-adaption neuron and random elastic deformation", Smart Struct. Syst., Int. J., 29(1), 1-16. https://doi.org/10.12989/sss.2022.29.1.001
  42. Zhou, S. and Song, W. (2020), "Concrete roadway crack segmentation using encoder-decoder networks with range images", Autom. Constr., 120, 103403. https://doi.org/10.1016/j.autcon.2020.103403
  43. Zhu, Q. (2016), "Pavement crack detection algorithm Based on image processing analysis", Proceedings in the 8th International Conference on Intelligent Human-Machine Systems and Cybernetics, IHMSC 2016, Vol. 1, pp. 15-18.