Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Investigation of the super-resolution methods for vision based structural measurement

  • Wu, Lijun (College of Physics and Information Engineering, Fuzhou University) ;
  • Cai, Zhouwei (College of Physics and Information Engineering, Fuzhou University) ;
  • Lin, Chenghao (College of Physics and Information Engineering, Fuzhou University) ;
  • Chen, Zhicong (College of Physics and Information Engineering, Fuzhou University) ;
  • Cheng, Shuying (College of Physics and Information Engineering, Fuzhou University) ;
  • Lin, Peijie (College of Physics and Information Engineering, Fuzhou University)
  • Received : 2022.06.26
  • Accepted : 2022.07.06
  • Published : 2022.09.25
⟨ Previous Next ⟩

Abstract

The machine-vision based structural displacement measurement methods are widely used due to its flexible deployment and non-contact measurement characteristics. The accuracy of vision measurement is directly related to the image resolution. In the field of computer vision, super-resolution reconstruction is an emerging method to improve image resolution. Particularly, the deep-learning based image super-resolution methods have shown great potential for improving image resolution and thus the machine-vision based measurement. In this article, we firstly review the latest progress of several deep learning based super-resolution models, together with the public benchmark datasets and the performance evaluation index. Secondly, we construct a binocular visual measurement platform to measure the distances of the adjacent corners on a chessboard that is universally used as a target when measuring the structure displacement via machine-vision based approaches. And then, several typical deep learning based super resolution algorithms are employed to improve the visual measurement performance. Experimental results show that super-resolution reconstruction technology can improve the accuracy of distance measurement of adjacent corners. According to the experimental results, one can find that the measurement accuracy improvement of the super resolution algorithms is not consistent with the existing quantitative performance evaluation index. Lastly, the current challenges and future trends of super resolution algorithms for visual measurement applications are pointed out.

Keywords

Acknowledgement

This work was supported in part by the Fujian Provincial Department of Science and Technology of China under Grant 2019H0006, 2021J01580 and 2022H0008; in part by the National Natural Science Foundation of China under Grant 51508105.

References

  1. Agustsson, E. and Timofte, R. (2017), "Ntire 2017 challenge on single image super-resolution: Dataset and study", Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, July, pp. 1122-1131.
  2. Ahn, N., Kang, B. and Sohn, K.A. (2018), "Fast, accurate, and lightweight super-resolution with cascading residual network", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 252-268.
  3. Arbelaez, P., Maire, M., Fowlkes, C. and Malik, J. (2011), "Contour detection and hierarchical image segmentation", IEEE Transact. Pattern Anal. Mach. Intell., 33(5), 898-916. https://doi.org/10.1109/TPAMI.2010.161
  4. Bevilacqua, M., Roumy, A., Guillemot, C. and Alberi-Morel, M.L. (2012), "Low-complexity single-image super-resolution based on nonnegative neighbor embedding", Proceedings of the 23rd British Machine Vision Conference (BMVC), Surrey, UK, September, pp. 1-10. http://dx.doi.org/10.5244/C.26.135
  5. Blu, T., Thevenaz, P. and Unser, M. (2004), "Linear interpolation revitalized", IEEE Transact. Image Process., 13(5), 710-719. https://doi.org/10.1109/TIP.2004.826093
  6. Bouguet, J.Y. (2013), Camera Calibration Toolbox for Matlab, Wwwvisioncaltechedu/bouguetj
  7. Bulat, A., Yang, J. and Tzimiropoulos, G. (2018), "To learn image super-resolution, use a GAN to learn how to do image degradation first", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 185-200.
  8. Caetano, E., Silva, S. and Bateira, J. (2007), "Application of a vision system to the monitoring of cable structures", Proceedings of the 7th International Symposium on Cable Dynamics, Vienna, Austria, December, pp. 225-236.
  9. Cai, J., Zeng, H., Yong, H., Cao, Z. and Zhang, L. (2019), "Toward real-world single image super-resolution: A new benchmark and a new model", Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, October-November, pp. 3086-3095.
  10. Cha, Y.J., Chen, J.G. and Buyukozturk, O. (2017), "Output-only computer vision based damage detection using phase-based optical flow and unscented Kalman filters", Eng. Struct., 132, 300-313. https://doi.org/10.1016/j.engstruct.2016.11.038
  11. Chang, H., Yeung, D.Y. and Xiong, Y. (2004), "Super-resolution through neighbor embedding", Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, Washington, DC, USA, June-July. https://doi.org/10.1109/CVPR.2004.1315043
  12. Chantas, G.K., Galatsanos, N.P. and Woods, N.A. (2007), "Superresolution based on fast registration and maximum a posteriori reconstruction", IEEE Transact. Image Process., 16(7), 1821-1830. https://doi.org/10.1109/TIP.2007.896664
  13. Chen, Y.W. (2011), "Learning-Based Super-Resolution", IEICE Technical Report; 111, 61-66.
  14. Dong, C., Loy, C.C., He, K. and Tang, X. (2014), "Learning a deep convolutional network for image super-resolution", Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland, September, pp. 184-199.
  15. Dong, C., Loy, C.C. and Tang, X. (2016), "Accelerating the superresolution convolutional neural network", Proceedings of European Conference on Computer Vision, Amsterdam, The Netherlands, October, pp. 391-407. https://doi.org/10.1007/978-3-319-46475-6_25
  16. Elad, M. and Feuer, A. (2002), "Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images", IEEE Transact. Image Process., 6(12), 1646-1658. https://doi.org/10.1109/83.650118
  17. Feng, D. and Feng, M.Q. (2015), "Model updating of railway bridge using in situ dynamic displacement measurement under trainloads", J. Bridge Eng., 20(12), 4015019. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000765
  18. Finkelstein, S.E., Schrump, D.S., Nguyen, D.M., Hewitt, S.M., Kunst, T.F. and Summers, R.M. (2003), "Comparative evaluation of super high-resolution CT scan and virtual bronchoscopy for the detection of tracheobronchial malignancies", Chest, 124(5), 1834-1840. https://doi.org/10.1378/chest.124.5.1834
  19. Freeman, W.T., Jones, T.R. and Pasztor, E.C. (2002), "Examplebased super-resolution", Comput. Graphics Applicat. IEEE, 22(2), 56-65. https://doi.org/10.1109/38.988747
  20. Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T. and Aizawa, K. (2016), "Manga109 dataset and creation of metadata", Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding, Cancun, Mexico, December, pp. 1-5. https://doi.org/10.1145/3011549.3011551
  21. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014), "Generative adversarial nets", (Ghahramani, Z., Welling, M., Cortes, C., et al.), In: Advances in Neural Information Processing Systems, Vol. 27, pp. 2672-2680.
  22. Hansen, R.S., Waldram, D.W., Thai, T.Q. and Berke, R.B. (2021), "Super Resolution Digital Image Correlation (SR-DIC): An Alternative to Image Stitching at High Magnifications", Experim. Mech., 61(9), 1351-1368. https://doi.org/10.1007/s11340-021-00729-2
  23. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 770-778.
  24. Hore, A. and Ziou, D. (2010), "Image quality metrics: PSNR vs. SSIM", Proceedings of 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, August, pp. 2366-2369. https://doi.org/10.1109/ICPR.2010.579
  25. Huang, J.B., Singh, A. and Ahuja, N. (2015), "Single image superresolution from transformed self-exemplars", Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June, pp. 5197-5206.
  26. Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K.Q. (2017), "Densely connected convolutional networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 2261-2269.
  27. Irani, M. and Peleg, S. (1991), "Improving resolution by image registration", CVGIP: Graphical models and image processing, 53(3), 231-239. https://doi.org/10.1016/1049-9652(91)90045-L
  28. Keys, R.G. (2003), "Cubic convolution interpolation for digital image processing", IEEE Transact. Acoust. Speech Signal Process., 29(6), 1153-1160. https://doi.org/10.1109/TASSP.1981.1163711
  29. Kim, J., Lee, J.K. and Lee, K.M. (2016a), "Accurate image superresolution using very deep convolutional networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1646-1654.
  30. Kim, J., Lee, J.K. and Lee, K.M. (2016b), "Deeply-recursive convolutional network for image super-resolution", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1637-1645.
  31. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012), "Imagenet classification with deep convolutional neural networks", Adv. Neural Inform. Process. Syst., 25.
  32. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. and Shi, W. (2017), "Photo-realistic single image super-resolution using a generative adversarial network", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 105-114.
  33. Lee, J.H., Jung, C.Y., Choi, E. and Cheung, J.H. (2017), "Visionbased multipoint measurement systems for structural in-plane and out-of-plane movements including twisting rotation", Smart Struct. Syst., Int. J., 20(5), 563-572. https://doi.org/10.12989/sss.2017.20.5.563
  34. Li, J., Fang, F., Mei, K. and Zhang, G. (2018), "Multi-scale residual network for image super-resolution", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 527-532.
  35. Li, B., Wang, B., Liu, J., Qi, Z. and Shi, Y. (2020), "s-LWSR: Super lightweight super-resolution network", IEEE Transact. Image Process., 29, 8368-8380. https://doi.org/10.1109/TIP.2020.3014953
  36. Lim, B., Son, S., Kim, H., Nah, S. and Mu Lee, K. (2017), "Enhanced deep residual networks for single image superresolution", Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July, pp. 136-144.
  37. Lin, Z., He, J., Tang, X. and Tang, C.K. (2008), "Limits of learning-based superresolution algorithms", Int. J. Comput. Vis., 80(3), 406-420. https://doi.org/10.1007/s11263-008-0148-2
  38. Liu, Y., Dong, Z., Lim, K.P. and Ling, N. (2020a), "A densely connected face super-resolution network based on attention mechanism", Proceedings of 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), Kristiansand, Norway, November, pp. 148-152. https://doi.org/10.1109/ICIEA48937.2020.9248111
  39. Liu, K., Ma, Y., Xiong, H., Yan, Z., Zhou, Z., Fang, P. and Liu, C. (2020b), "Medical image super-resolution method based on dense blended attention network", Laser & Optoelectronics Progress, 57(2), 021014. https://doi.org/10.48550/arXiv.1905.05084
  40. Luo, Y., Zhou, L., Wang, S. and Wang, Z. (2017), "Video satellite imagery super resolution via convolutional neural networks", IEEE Geosci. Remote Sens. Lett., 14(12), 2398-2402. https://doi.org/10.1109/LGRS.2017.2766204
  41. Martin, D., Fowlkes, C., Tal, D. and Malik, J. (2001), "A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics", Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV 2001), Vol. 2, Vancouver, BC, Canada, July, pp. 416-423. https://doi.org/10.1109/ICCV.2001.937655
  42. Mecheri, K., Ziou, D. and Deschenes, F. (2007), "Super-resolution based on interpolation and global sub pixel translation", Proceedings of the 4th Canadian Conference on Computer and Robot Vision (CRV'07), Montreal, QC, Canada, May, pp. 448-458. https://doi.org/10.1109/CRV.2007.62
  43. Nguyen, M.Q., Atkinson, P.M. and Lewis, H.G. (2005), "Superresolution mapping using a Hopfield neural network with LIDAR data", IEEE Geosci. Remote Sens. Lett., 2(3), 366-370. https://doi.org/10.1109/LGRS.2005.851551
  44. Ni, K.S. and Nguyen, T.Q. (2007), "Image superresolution using support vector regression", IEEE Trans. Image Process., 16(6), 1596-1610. https://doi.org/10.1109/TIP.2007.896644
  45. Oh, B.K., Hwang, J.W., Kim, Y., Cho, T. and Park, H.S. (2015), "Vision-based system identification technique for building structures using a motion capture system", J. Sound Vib., 356, 72-85. https://doi.org/10.1016/j.jsv.2015.07.011
  46. Ojio, T., Carey, C., O'Brien, E.J., Doherty, C. and Taylor, S.E. (2016), "Contactless bridge weigh-in-motion", J. Bridge Eng., 21, 4016032. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000776
  47. Park, S.C., Park, M.K. and Kang, M.G. (2003), "Super-resolution image reconstruction: a technical overview", IEEE Signal Process. Magaz., 20(3), 21-36. https://doi.org/10.1109/MSP.2003.1203207
  48. Park, S.J., Son, H., Cho, S., Hong, K.S. and Lee, S. (2018), "Srfeat: Single image super-resolution with feature discrimination", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 439-455.
  49. Rasti, P., Uiboupin, T., Escalera, S. and Anbarjafari, G. (2016), "Convolutional neural network super resolution for face recognition in surveillance monitoring", Proceedings of International Conference on Articulated Motion and Deformable Objects, Palma de Mallorca, Spain, July, pp. 175-184. https://doi.org/10.1007/978-3-319-41778-3_18
  50. Schultz, R.R. and Stevenson, R.L. (1996), "Extraction of highresolution frames from video sequences", IEEE Transact. Image Process. A Publicat. IEEE Signal Process. Soc., 5(6), 996-1011. https://doi.org/10.1109/83.503915
  51. Seif, G. and Androutsos, D (2018), "Edge-based loss function for single image super-resolution", Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, Alberta, Canada, April, pp. 1468-1472. https://doi.org/10.1109/ICASSP.2018.8461664
  52. Sheikh, H.R., Bovik, A.C. and De Veciana, G. (2006), "An information fidelity criterion for image quality assessment using natural scene statistics", IEEE Transact. Image Process., 14(12), 2117-2128. https://doi.org/10.1109/TIP.2005.859389
  53. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D. and Wang, Z. (2016), "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1874-1883.
  54. Song, X., Dai, Y. and Qin, X. (2016), "Deep depth superresolution: Learning depth super-resolution using deep convolutional neural network", Proceedings of the 13th Asian Conference on Computer Vision, Taipei, Taiwan, November, pp. 360-376. https://doi.org/10.1007/978-3-319-54190-7_22
  55. Song, X., Liu, W., Liu, J., Liu, C., Lu, C. and Gao, H. (2019), "Deep CNN jointing low-high level feature for image superresolution", Proceedings of the 10th International Conference on Graphics and Image Processing (ICGIP 2018), Vol. 11069, Chengdu, China, December, pp. 1059-1065. https://doi.org/10.1117/12.2524412
  56. Stark, H. and Oskoui, P. (1989), "High-resolution image recovery from image-plane arrays, using convex projections", J. Optical Soc. Am. A Optics Image Sci., 6(11), 1715-1726. https://doi.org/10.1364/JOSAA.6.001715
  57. Tai, Y., Yang, J., Liu, X. and Xu, C. (2017a), "Memnet: A persistent memory network for image restoration", Proceedings of the IEEE International Conference on Computer Vision, Salt Lake City, UT, USA, June, pp. 4539-4547.
  58. Tai, Y., Yang, J. and Liu, X. (2017b), "Image super-resolution via deep recursive residual network", Proceedings of the 30th 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July, pp. 2790-2798.
  59. Tian, J. and Ma, K.K. (2011), "A survey on super-resolution imaging", Signal Image Video Process., 5(3), 329-342. https://doi.org/10.1007/s11760-010-0204-6
  60. Tong, T., Li, G., Liu, X. and Gao, Q. (2017), "Image superresolution using dense skip connections", Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, October, pp. 4809-4817.
  61. Trajkovic, M. and Hedley, M (1998), "Fast corner detection", Image Vision Comput., 16(2), 75-87. https://doi.org/10.1016/S0262-8856(97)00056-5
  62. Wang, Z., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P. (2004), "Image quality assessment: from error visibility to structural similarity", IEEE Trans. Image Process., 13(4), 600-612. https://doi.org/10.1109/TIP.2003.819861
  63. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y. and Change Loy, C. (2018), "Esrgan: Enhanced super-resolution generative adversarial networks", Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, September.
  64. Wang, Z., Chen, J. and Hoi, S.C. (2020a), "Deep learning for image super-resolution: A survey", IEEE Transact. Pattern Anal. Mach. Intell., 43(10), 3365-3387. https://doi.org/10.1109/TPAMI.2020.2982166
  65. Wang, W., Hu, Y., Luo, Y. and Zhang, T. (2020b), "Brief survey of single image super-resolution reconstruction based on deep learning approaches", Sens. Imag., 21(1), 1-20. https://doi.org/10.1007/s11220-020-00285-4
  66. Wu, L.J., Casciati, F. and Casciati, S. (2014), "Dynamic testing of a laboratory model via vision-based sensing", Eng. Struct., 60, 113-125. https://doi.org/10.1016/j.engstruct.2013.12.002
  67. Wu, L., Su, Y., Chen, Z., Chen, S., Cheng, S. and Lin, P. (2020a), "Six-degree-of-freedom generalized displacements measurement based on binocular vision", Struct. Control Health Monitor., 27(2), e2458. https://doi.org/10.1002/stc.2458
  68. Wu, Q., Fan, C., Li, Y., Li, Y. and Hu, J. (2020b), "A novel perceptual loss function for single image super-resolution", Multimedia Tools Applicat., 79(29), 21265-21278. https://doi.org/10.1007/s11042-020-08878-7
  69. Xu, Y. and Brownjohn, J.M. (2018), "Review of machine-vision based methodologies for displacement measurement in civil structures", J. Civil Struct. Health Monitor., 8, 91-110. https://doi.org/10.1007/s13349-017-0261-4
  70. Yan, X., Shen, Q. and Liu, X. (2015), "Super-resolution reconstruction for license plate image in video surveillance system", Proceedings of 2015 10th International Conference on Communications and Networking in China (ChinaCom), Shanghai, China, August, pp. 643-647. https://doi.org/10.1109/CHINACOM.2015.7498016
  71. Yang, J., Wright, J., Huang, T.S. and Ma, Y. (2010), "Image superresolution via sparse representation", IEEE Transact. Image Process., 19(11), 2861-2873. https://doi.org/10.1109/TIP.2010.2050625
  72. Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.H. and Liao, Q. (2019a), "Deep learning for single image super-resolution: A brief review", IEEE Transact. Multimedia, 21(12), 3106-3121. https://doi.org/10.1109/TMM.2019.2919431
  73. Yang, W., Wang, W., Zhang, X., Sun, S. and Liao, Q. (2019b), "Lightweight feature fusion network for single image superresolution", IEEE Signal Process. Lett., 26(4), 538-542. https://doi.org/10.1109/LSP.2018.2890770
  74. Ye, X.W., Yi, T.H., Dong, C.Z. and Liu, T. (2016a), "Vision-based structural displacement measurement: System performance evaluation and influence factor analysis", Measurement, 88, 372-384. https://doi.org/10.1016/j.measurement.2016.01.024
  75. Ye, X.W., Dong, C.Z. and Liu, T. (2016b), "Image-based structural dynamic displacement measurement using different multi-object tracking algorithms", Smart Struct. Syst., Int. J., 17(6), 935-956. https://doi.org/10.12989/sss.2016.17.6.935
  76. Yoon, H., Elanwar, H., Choi, H., Golparvar-Fard, M. and Spencer Jr, B.F. (2016), "Target-free approach for vision-based structural system identification using consumergrade cameras", Struct. Control Health Monitor., 23(12), 1405-1416. https://doi.org/10.1002/stc.1850
  77. Yu, J., Fan, Y., Yang, J., Xu, N., Wang, Z., Wang, X. and Huang, T. (2018), "Wide activation for efficient and accurate image superresolution", arXiv preprint arXiv:1808.08718. https://doi.org/10.48550/arXiv.1808.08718
  78. Zeyde, R., Elad, M. and Protter, M. (2010), "On single image scale-up using sparse-representations", Proceedings of International Conference on Curves and Surfaces, Avignon, France, June, pp. 711-730. https://doi.org/10.1007/978-3-642-27413-8_47
  79. Zhang, Z. (2000), "A flexible new technique for camera calibration", IEEE Transact. Pattern Anal. Mach. Intell., 22(11), 1330-1334. https://doi.org/10.1109/34.888718
  80. Zhang, K., Zuo, W. and Zhang, L. (2017), "Learning a single convolutional super-resolution network for multiple degradations", Proceedings of Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 33662-3271.
  81. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B. and Fu, Y. (2018a), "Image super-resolution using very deep residual channel attention networks", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 286-301.
  82. Zhang, Y., Tian, Y., Kong, Y., Zhong, B. and Fu, Y. (2018b), "Residual dense network for image super-resolution", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June, pp. 2472-2481.
  83. Zhang, Y., Liu, S., Dong, C., Zhang, X. and Yuan, Y. (2019), "Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution", IEEE Transact. Image Process., 29, 1101-1112. https://doi.org/10.1109/TIP.2019.2938347