Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

A new image-quality evaluating and enhancing methodology for bridge inspection using an unmanned aerial vehicle

  • Lee, Jin Hwan (Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology) ;
  • Yoon, Sungsik (Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign) ;
  • Kim, Byunghyun (Department of Civil Engineering, University of Seoul) ;
  • Gwon, Gi-Hun (Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology) ;
  • Kim, In-Ho (Department of Civil Engineering, Kunsan National University) ;
  • Jung, Hyung-Jo (Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology)
  • Received : 2020.10.09
  • Accepted : 2020.12.15
  • Published : 2021.02.25
⟨ Previous Next ⟩

Abstract

This paper proposes a new methodology to address the image quality problem encountered as the use of an unmanned aerial vehicle (UAV) in the field of bridge inspection increased. When inspecting a bridge, the image obtained from the UAV was degraded by various interference factors such as vibration, wind, and motion of UAV. Image quality degradation such as blur, noise, and low-resolution is a major obstacle in utilizing bridge inspection technology based on UAV. In particular, in the field of bridge inspection where damages must be accurately and quickly detected based on data obtained from UAV, these quality issues weaken the advantage of using UAVs by requiring re-take of images through re-flighting. Therefore, in this study, image quality assessment (IQA) based on local blur map (LBM) and image quality enhancement (IQE) using the variational Dirichlet (VD) kernel estimation were proposed as a solution to address the quality issues. First, image data was collected by setting different camera parameters for each bridge member. Second, a blur map was generated through discrete wavelet transform (DWT) and a new quality metric to measure the degree of blurriness was proposed. Third, for low-quality images with a large degree of blurriness, the blind kernel estimation and blind image deconvolution were performed to enhance the quality of images. In the validation tests, the proposed quality metric was applied to material image sets of bridge pier and deck taken from UAV, and its results were compared with those of other quality metrics based on singular value decomposition (SVD), sum of gray-intensity variance (SGV) and high-frequency multiscale fusion and sort transform (HiFST) methods. It was validated that the proposed IQA metric showed better classification performance on UAV images for bridge inspection through comparison with the classification results by human perception. In addition, by performing IQE, on average, 26% of blur was reduced, and the images with enhanced quality showed better damage detection performance through the deep learning model (i.e., mask and region-based convolutional neural network).

Keywords

Acknowledgement

This research was financially funded by Construction Technology Research Program funded by Ministry of Land, Infrastructure and Transport (MOLIT) of Korean government (grant number 19SCIP-C116873-04). This work is also financially supported by Korea Ministry of Land, Infrastructure and Transport (MOLIT) as 「Innovative Talent Education Program for Smart City」.

References

  1. Burdziakowski, P. (2020), "A novel method for the deblurring of photogrammetric images using conditional generative adversarial networks", Remote Sensing, 12(16), 2586. https://doi.org/10.3390/rs12162586
  2. Cho, S. and Lee, S. (2009), "Fast motion deblurring", ACM SIGGRAPH Asia 2009 papers, pp. 1-8. https://doi.org/10.1145/1661412.1618491
  3. Cunningham, C.S., Ransom, D., Wilkes, J., Bishop, J. and White, B. (2015), "Mechanical Design Features of a Small Gas Turbine for Power Generation in Unmanned Aerial Vehicles", Proceedings of Turbo Expo: Power for Land, Sea, and Air, 56796, V008T23A021.
  4. Dong, C., Loy, C.C. and Tang, X. (2016), "Accelerating the superresolution convolutional neural network", Proceedings of European Conference on Computer Vision, pp. 391-407.
  5. Dorafshan, S., Maguire, M., Hoffer, N.V. and Coopmans, C. (2017), "Challenges in bridge inspection using small unmanned aerial systems: Results and lessons learned", Proceedings of the 2017 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1722-1730.
  6. Duque, L., Seo, J. and Wacker, J. (2018), "Bridge deterioration quantification protocol using UAV", J. Bridge Eng., 23(10), 04018080. https://doi.org/(ASCE)BE.1943-5592.0001289 https://doi.org/10.1061/(asce)be.1943-5592.0001289
  7. Gao, J., Liao, W., Nuyttens, D., Lootens, P., Vangeyte, J., Pizurica, A. and Pieters, J.G. (2018), "Fusion of pixel and object-based features for weed mapping using unmanned aerial vehicle imagery", Int. J. Appl. Earth Observ. Geoinform., 67, 43-53. https://doi.org/10.1016/j.jag.2017.12.012
  8. Golestaneh, S.A. and Karam, L.J. (2017), "Spatially-varying blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes", CVPR 2017, pp. 596-605.
  9. Hallermann, N. and Morgenthal, G. (2014), "Visual inspection strategies for large bridges using Unmanned Aerial Vehicles (UAV)", Proceedings of the 7th IABMAS, International Conference on Bridge Maintenance, Safety and Management, pp. 661-667.
  10. He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017), "Mask RCNN", Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969.
  11. Jeong, G.Y., Nguyen, T.N., Tran, D.K. and Hoang, T.B.H. (2020), "Applying unmanned aerial vehicle photogrammetry for measuring dimension of structural elements in traditional timber building", Measurement, 153, 107386. https://doi.org/10.1016/j.measurement.2019.107386
  12. Jordan, S., Moore, J., Hovet, S., Box, J., Perry, J., Kirsche, K. and Tse, Z.T.H. (2018), "State-of-the-art technologies for UAV inspections", IET Radar, Sonar & Navigation, 12(2), 151-164. https://doi.org/10.1049/iet-rsn.2017.0251
  13. Jung, H.J., Lee, J.H., Yoon, S. and Kim, I.H. (2019), "Bridge inspection and condition assessment using unmanned aerial vehicles (UAVs): major challenges and solutions from a practical perspective", Smart Struct. Syst., Int. J., 24(5), 669-681. https://doi.org/10.12989/sss.2019.24.5.669
  14. Kim, B. and Cho, S. (2019), "Image-based concrete crack assessment using mask and region-based convolutional neural network", Struct. Control Health Monitor., 26(8), e2381. https://doi.org/10.1002/stc.2381
  15. Kim, J., Oh, J. and Park, R.H. (2016), "Removing non-uniform camera shake using blind motion deblurring", Proceedings of the 2016 IEEE International Conference on Consumer Electronics (ICCE), pp. 351-352.
  16. Kim, I.H., Jeon, H., Baek, S.C., Hong, W.H. and Jung, H.J. (2018), "Application of crack identification techniques for an aging concrete bridge inspection using an unmanned aerial vehicle", Sensors, 18(6), 1881. https://doi.org/10.3390/s18061881
  17. Lei, J., Zhang, S., Luo, L., Xiao, J. and Wang, H. (2018), "Superresolution enhancement of UAV images based on fractional calculus and POCS", Geo-spatial Inform. Sci., 21(1), 56-66. https://doi.org/10.1080/10095020.2018.1424409
  18. Liu, L., Liu, B., Huang, H. and Bovik, A.C. (2014a), "Noreference image quality assessment based on spatial and spectral entropies", Signal Process.: Image Commun., 29(8), 856-863. https://doi.org/10.1016/j.image.2014.06.006
  19. Liu, Q.F., Xiao, S.F., Huang, K.Z. and Zhong, Z. (2014b), "A SVD-based optical MIMO precoding scheme in indoor visible light communication", Int. J. Future Comput. Commun., 3(6), 421. https://doi.org/10.7763/IJFCC.2014.V3.340
  20. Liu, H., Wang, W., He, Z., Tong, Q., Wang, X., Yu, W. and Lv, M. (2015), "Blind image quality evaluation metrics design for UAV photographic application", Proceedings of IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 293-297.
  21. Liu, Y., Yeoh, J.K. and Chua, D.K. (2020), "Deep learning-based enhancement of motion blurred UAV concrete crack images", J. Comput. Civil Eng., 34(5), 04020028. https://doi.org/(ASCE)CP.1943-5487.0000907 https://doi.org/10.1061/(ASCE)CP.1943-5487.0000907
  22. Mittal, A., Moorthy, A.K. and Bovik, A.C. (2012a), "No-reference image quality assessment in the spatial domain", IEEE Transactions on Image Processing, 21(12), 4695-4708. https://doi.org/ 10.1109/TIP.2012.2214050
  23. Mittal, A., Soundararajan, R. and Bovik, A.C. (2012b), "Making a "completely blind" image quality analyzer", IEEE Signal Processing Letters, 20(3), 209-212. https://doi.org/ 10.1109/LSP.2012.2227726
  24. Morgenthal, G. and Hallermann, N. (2014), "Quality assessment of unmanned aerial vehicle (UAV) based visual inspection of structures", Adv. Struct. Eng., 17(3), 289-302. https://doi.org/10.1260/1369-4332.17.3.289
  25. Morgenthal, G., Hallermann, N., Kersten, J., Taraben, J., Debus, P., Helmrich, M. and Rodehorst, V. (2019), "Framework for automated UAS-based structural condition assessment of bridges", Automat. Constr., 97, 77-95. https://doi.org/10.1016/j.autcon.2018.10.006
  26. Myeong, W. and Myung, H. (2018), "Development of a wallclimbing drone capable of vertical soft landing using a tilt-rotor mechanism", IEEE Access, 7, 4868-4879. https://doi.org/10.1109/ACCESS.2018.2889686
  27. O'Connor, J., Smith, M.J. and James, M.R. (2017), "Cameras and settings for aerial surveys in the geosciences: Optimising image data", Progress Phys. Geography, 41(3), 325-344. https://doi.org/10.1177/0309133317703092
  28. Salaan, C.J.O., Okada, Y., Mizutani, S., Ishii, T., Koura, K., Ohno, K. and Tadokoro, S. (2018), "Close visual bridge inspection using a UAV with a passive rotating spherical shell", J. Field Robot., 35(6), 850-867. https://doi.org/10.1002/rob.21781
  29. Seo, J., Duque, L. and Wacker, J. (2018), "Drone-enabled bridge inspection methodology and application", Automat. Constr., 94, 112-126. https://doi.org/10.1016/j.autcon.2018. 06.006
  30. Sieberth, T., Wackrow, R. and Chandler, J.H. (2015), "UAV image blur - its influence and ways to correct it", The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 40(1), 33. https://doi.org/10.5194/isprsarchives-XL-1-W4-33-2015
  31. Sieberth, T., Wackrow, R. and Chandler, J.H. (2016), "Automatic detection of blurred images in UAV image sets", ISPRS J. Photogrammetry Remote Sens., 122, 1-16. https://doi.org/10.1016/j.isprsjprs.2016.09.010
  32. Su, B., Lu, S. and Tan, C.L. (2011), "Blurred image region detection and classification", Proceedings of the 19th ACM International Conference on Multimedia, pp. 1397-1400.
  33. Tomiczek, A.P., Bridge, J.A., Ifju, P.G., Whitley, T.J., Tripp, C.S., Ortega, A.E. and Gonzalez, S.A. (2018), "Small unmanned aerial vehicle (sUAV) inspections in GPS denied area beneath bridges", Proceedings of the Structures Congress 2018: Bridges, Transportation Structures, and Nonbuilding Structures, pp. 205-216.
  34. Wang, R., Ma, G., Qin, Q., Shi, Q. and Huang, J. (2018), "Blind UAV images deblurring based on discriminative networks", Sensors, 18(9), 2874. https://doi.org/10.3390/s18092874
  35. Xie, S., Girshick, R., Dollar, P., Tu, Z. and He, K. (2017), "Aggregated residual transformations for deep neural networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492-1500.
  36. Yoon, S., Gwon, G.H., Lee, J.H. and Jung, H.J. (2020), "Three-dimensional image coordinate-based missing region of interest area detection and damage localization for bridge visual inspection using unmanned aerial vehicles", Struct. Health Monitor., 1475921720918675. https://doi.org/10.1177/1475921720918675
  37. Zhou, X., Mateos, J., Zhou, F., Molina, R. and Katsaggelos, A.K. (2015), "Variational Dirichlet blur kernel estimation", IEEE Transact. Image Process., 24(12), 5127-5139. https://doi.org/10.1109/TIP.2015.2478407