Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Autonomous pothole detection using deep region-based convolutional neural network with cloud computing

  • Luo, Longxi (Department of Civil Engineering, Tsinghua University) ;
  • Feng, Maria Q. (Department of Civil Engineering and Engineering Mechanics, Columbia University) ;
  • Wu, Jianping (Department of Civil Engineering, Tsinghua University) ;
  • Leung, Ryan Y. (Department of Civil Engineering and Engineering Mechanics, Columbia University)
  • Received : 2019.05.26
  • Accepted : 2019.08.06
  • Published : 2019.12.25
⟨ Previous Next ⟩

Abstract

Road surface deteriorations such as potholes have caused motorists heavy monetary damages every year. However, effective road condition monitoring has been a continuing challenge to road owners. Depth cameras have a small field of view and can be easily affected by vehicle bouncing. Traditional image processing methods based on algorithms such as segmentation cannot adapt to varying environmental and camera scenarios. In recent years, novel object detection methods based on deep learning algorithms have produced good results in detecting typical objects, such as faces, vehicles, structures and more, even in scenarios with changing object distances, camera angles, lighting conditions, etc. Therefore, in this study, a Deep Learning Pothole Detector (DLPD) based on the deep region-based convolutional neural network is proposed for autonomous detection of potholes from images. About 900 images with potholes and road surface conditions are collected and divided into training and testing data. Parameters of the network in the DLPD are calibrated based on sensitivity tests. Then, the calibrated DLPD is trained by the training data and applied to the 215 testing images to evaluate its performance. It is demonstrated that potholes can be automatically detected with high average precision over 93%. Potholes can be differentiated from manholes by training and applying a manhole-pothole classifier which is constructed using the convolutional neural network layers in DLPD. Repeated detection of the same potholes can be prevented through feature matching of the newly detected pothole with previously detected potholes within a small region.

Keywords

Acknowledgement

Supported by : National Natural Science Foundation of China

References

  1. ASCE (2017), "2017 infrastructure report card", https://www.infrastructurereportcard.org/cat-item/roads.
  2. Bousquet, O. and Bottou, L. (2008). "The tradeoffs of large scale learning", Proc. of NIPS 2008.
  3. Cha, Y.J., Chen, J. and Buyukozturk, O. (2017), "Output-only computer vision based damage detection using phase-based optical flow and unscented Kalman filters", Eng. Struct., 132, 300-313. https://doi.org/10.1016/j.engstruct.2016.11.038.
  4. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2017), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput-Aided Civ. Inf., 33, 731-747. https://doi.org/10.1111/mice.12334.
  5. Chen, J.G., Wadhwa, N., Cha, Y.J., Durand, F., Freeman, W.T. and Buyukozturk, O. (2015), "Modal identification of simple structures with high-speed video using motion magnification", J. Sound Vib., 345, 58-71. https://doi.org/10.1016/j.jsv.2015.01.024.
  6. Chen, Y.L., Jahanshahi, M.R., Manjunatha, P., Gan, W., Abdelbarr, M., Masri, S.F., Becerik-Gerber, B. and Caffrey, J.P. (2016), "Inexpensive multimodal sensor fusion system for autonomous data acquisition of road surface conditions", IEEE Sen. J., 16(21), 7731-7743. DOI: 10.1109/JSEN.2016.2602871
  7. Cord, A. and Chambon, S. (2012), "Automatic road defect detection by textural pattern recognition based on AdaBoost", Comput-Aided Civ. Inf., 27(4), 244-259. https://doi.org/10.1111/j.1467-8667.2011.00736.x.
  8. Debella-Gilo, M. and Kaab, A. (2011), "Sub-pixel precision image matching for measuring surface displacements on mass movements using normalized cross-correlation", Remote Sens. Environ., 115(1), 130-142. https://doi.org/10.1016/j.rse.2010.08.012.
  9. Eriksson, J., Girod, L., Hull, B., Newton, R., Madden, S. and Balakrishnan, H. (2008), "The pothole patrol: using a mobile sensor network for road surface monitoring", Proc. of MobiCASE 2014.
  10. Fernandez-Llorca, D., Minguez, R.Q., Alonso, I.P., Lopez, C.F., Daza, I.G., Sotelo, M.A. and Cordero, C.A. (2017), "Assistive intelligent transportation systems: the need for user localization and anonymous disability identification", IEEE Intel. Transp. Sy., 9(2), 25-40. DOI: 10.1109/MITS.2017.2666579.
  11. Girshick, R. (2015), "Fast R-CNN", Proc. of IEEE ICCV 2015.
  12. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich feature hierarchies for accurate object detection and semantic segmentation", Proc. of IEEE CVPR 2014.
  13. Greenbaum, R.J., Smyth, A.W. and Chatzis, M.N. (2015), "Monocular computer vision method for the experimental study of three-dimensional rocking motion", J. Eng. Mech. - ASCE., 142(1), 04015062. https://doi.org/10.1061/(ASCE)EM.1943-7889.0000972.
  14. Jahanshahi, M.R., Chen, F.C., Ansar, A., Padgett, C.W., Clouse, D. and Bayard, D.S. (2017), "Accurate and robust scene reconstruction in the presence of misassociated features for aerial sensing", J. Comput. Civil Eng., 31(6), 04017056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000702.
  15. Jahanshahi, M.R., Jazizadeh, F., Masri, S.F. and Becerik-Gerber, B. (2012), "Unsupervised approach for autonomous pavementdefect detection and quantification using an inexpensive depth sensor", J. Comput. Civil Eng., 27(6), 743-754. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000245.
  16. Jang, J., Yang, Y., Smyth, A.W., Cavalcanti, D. and Kumar, R. (2016), "Framework of data acquisition and integration for the detection of pavement distress via multiple vehicles", J. Comput. Civil Eng., 31(2), 04016052. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000618.
  17. Jog, G., Koch, C., Golparvar-Fard, M. and Brilakis, I. (2012), "Pothole properties measurement through visual 2D recognition and 3D reconstruction", Proc. of ASCE 2012.
  18. Kalal, Z., Mikolajczyk, K. and Matas, J. (2010). "Forwardbackward error: Automatic detection of tracking failures", Proc. of ICPR 2010.
  19. Kim, S.W., Jeon, B.-G., Kim, N.-S. and Park, J.-C. (2013), "Vision-based monitoring system for evaluating cable tensile forces on a cable-stayed bridge", Struct. Health Monit., 12(5-6), 440-456. https://doi.org/10.1177/1475921713500513.
  20. Koch, C. and Brilakis, I. (2011), "Pothole detection in asphalt pavement images", Adv. Eng. Inform., 25(3), 507-515. https://doi.org/10.1016/j.aei.2011.01.002.
  21. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012). "Imagenet classification with deep convolutional neural networks", Proc. of ICONIP 2012.
  22. Lucas, B.D. and Kanade, T. (1981), "An iterative image registration technique with an application to stereo vision", Proc. of IJCAI 1981.
  23. Luo, L., Feng, M., Fukuda, Y. and Zhang, C. (2015), "Micro displacement and strain detection for crack prediction on concrete surface using optical nondestructive evaluation methods", Int. J. Progn. Health Manag., 6(SP3), 1-12.
  24. Luo, L., Feng, M.Q. and Wu, Z.Y. (2018.), "Robust vision sensor for multi-point displacement monitoring of bridges in the field", Eng. Struct., 163, 255-266. https://doi.org/10.1016/j.engstruct.2018.02.014.
  25. Mednis, A., Strazdins, G., Zviedris, R., Kanonirs, G. and Selavo, L. (2011). "Real time pothole detection using android smartphones with accelerometers", Proc. of DCOSS 2011.
  26. Park, K., Torbol, M. and Kim, S. (2018), "Vision-based natural frequency identification using laser speckle imaging and parallel computing", Comput-Aided Civ. Inf., 33(1), 51-63. https://doi.org/10.1111/mice.12312.
  27. Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster r-cnn:Towards real-time object detection with region proposal networks", Proc. of ICONIP 2015.
  28. Rosten, E. and Drummond, T. (2005), "Fusing points and lines for high performance tracking", Proc. of ICCV 2005.
  29. Rosten, E., Porter, R. and Drummond, T. (2008), "Faster and better: A machine learning approach to corner detection", IEEE T. Pattern Anal., 32(1), 105-119. DOI:10.1109/TPAMI.2008.275.
  30. Uijlings, J.R., Van De Sande, K.E., Gevers, T. and Smeulders, A.W. (2013), "Selective search for object recognition", Int. J. Comput. Vision., 104(2), 154-171. https://doi.org/10.1007/s11263-013-0620-5.
  31. Ye, X.W., Ni, Y.Q., Wai, T.T., Wong, K.Y., Zhang, X.M. and Xu, F. (2013), "A vision-based system for dynamic displacement measurement of long-span bridges: algorithm and verification", Smart Struct. Syst., 12(3-4), 363-379. http://dx.doi.org/10.12989/sss.2013.12.3_4.363.
  32. Ye, X.W., Dong, C.Z. and Liu, T. (2016a), "Force monitoring of steel cables using vision-based sensing technology:methodology and experimental verification", Smart Struct. Syst., 18(3), 585-599. http://dx.doi.org/10.12989/sss.2016.18.3.585
  33. Ye, X.W., Dong, C.Z. and Liu, T. (2016b), "Image-based structural dynamic displacement measurement using different multi-object tracking algorithms", Smart Struct. Syst., 17(6), 935-956. http://dx.doi.org/10.12989/sss.2016.17.6.935.
  34. Yoon, H., Shin, J. and Spencer, B.F. (2018), "Structural displacement measurement using an unmanned aerial system", Comput-Aided Civ. Inf., 33(3), 183-192. https://doi.org/10.1111/mice.12338