Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Feature-based image stitching for panorama construction and visual inspection of structures

  • Cheng, Kai (Department of Disaster Mitigation for Structures, Tongji University) ;
  • Shan, Jiazeng (Department of Disaster Mitigation for Structures, Tongji University) ;
  • Liu, Yuwen (Department of Disaster Mitigation for Structures, Tongji University)
  • Received : 2020.11.02
  • Accepted : 2021.08.09
  • Published : 2021.11.25
⟨ Previous Next ⟩

Abstract

This study presents a feature-based image stitching method with multi-level constraint criterion for panorama construction and visual inspection of building structures. The comparison of global view and local resolution over building exterior is discussed regarding practical implementation. An inspection-oriented methodology framework with optimized inlier distribution is designed for generating a feasible and reliable building panorama by using ordinary optic images. Two illustrative examples, including an earthquake-damaged masonry wall and a high-rise building with stone curtain walls, are experimentally investigated. The severely developed structural crack is fully mapped with stitched image and extracted in preparation for further quality evaluation. The curtain wall of the high-rise building is successfully constructed by using UAV-based images. The panorama quality is further compared with commercial stitching software and several improvements are illustrated in the particular case. In addition, the reliability of the proposed feature-based stitching approach is parametrically studied with different setups of input images.

Keywords

Acknowledgement

This study is sponsored by the National Natural Science Foundation of China (Grant No: 51878483), the Key Laboratory of Shock and Vibration of Engineering Materials and Structures, Sichuan Province (No. 19kfgk03), the Shanghai Qi Zhi Institute (Grant No. SYXF0120020109), and the Peak Discipline Construction Project of Shanghai (No. 2021-CE-03).

References

  1. Adobe Photoshop (2006), Create and edit a panorama in Photoshop, Adobe Photoshop, San Jose, USA. https://helpx.adobe.com/photoshop/using/create-panoramicimages-photomerge.html
  2. Agdas, D., Rice, J.A., Martinez, J.R. and Lasa, I.R. (2016), "Comparison of visual inspection and structural-health monitoring as bridge condition assessment methods", J. Perform. Constr Facil., 30(3), 04015049. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000802
  3. Akbar, M.A., Qidwai, U. and Jahanshahi, M.R. (2018), "An evaluation of image-based structural health monitoring using integrated unmanned aerial vehicle platform", Struct. Control Health Monitor., 26(1), e2276. https://doi.org/10.1002/stc.2276
  4. Bang, S., Kim, H. and Kim, H. (2017), "UAV-based automatic generation of high-resolution panorama at a construction site with a focus on preprocessing for image stitching", Automat. Constr., 84, 70-80. https://doi.org/10.1016/j.autcon.2017.08.031
  5. Bay, H., Ess, A., Tuytelaars, T. and Van Gool, L. (2008), "Speeded-up robust features (SURF)", Comput. Vis. Image Underst., 110(3), 346-359. https://doi.org/10.1016/j.cviu.2007.09.014
  6. Brown, M. and Lowe, D.G. (2007), "Automatic panoramic image stitching using invariant features", Int. J. Comput. Vis., 74(1), 59-73. https://doi.org/10.1007/s11263-006-0002-3
  7. Choi, J., Yeum, C.M., Dyke, S.J. and Jahanshahi, M.R. (2018), "Computer-aided approach for rapid post-event visual evaluation of a building facade", Sensors, 18(9), 3017. https://doi.org/10.3390/s18093017
  8. Fischler, M.A. and Bolles, R.C. (1981), "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Commun. ACM, 24(6), 381-395. https://doi.org/10.1145/358669.358692
  9. Harris, C.G. and Stephens, M. (1988), "A combined corner and edge detector", Alvey vision conference, Manchester, UK, September.
  10. Herrmann, C., Wang, C., Bowen, R.S., Keyder, E., Krainin, M., Liu, C. and Zabih, R. (2018), "Robust image stitching with multiple registrations", Proceedings of European Conference on Computer Vision, Munich, Germany, September.
  11. Hoskere, V., Park, J.W., Yoon, H. and Spencer, B.F. (2019), "Vision-based modal survey of civil infrastructure using unmanned aerial vehicles", J. Struct. Eng., 145(7), 04019062. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002321
  12. Hsu, T.Y., Pham, Q.V., Chao, W.C. and Yang, Y.S. (2020), "Post-earthquake building safety evaluation using consumer-grade surveillance cameras", Smart Struct. Syst., Int. J., 25(5), 531-541. https://doi.org/10.12989/sss.2020.25.5.531
  13. Iacovino, C., Ditommaso, R., Ponzo, F.C. and Limongelli, M.P. (2018), "The interpolation evolution method for damage localization in structures under seismic excitation", Earthq. Eng. Struct. Dyn., 47(10), 2117-2136. https://doi.org/10.1002/eqe.3062
  14. Jahanshahi, M.R., Masri, S.F. and Sukhatme, G.S. (2011), "Multi-image stitching and scene reconstruction for evaluating defect evolution in structures", Struct. Health Monitor., 10(6), 643-657. https://doi.org/10.1177/1475921710395809
  15. Jiang, X., Ma, J., Jiang, J. and Guo, X. (2020), "Robust feature matching using spatial clustering with heavy outliers", IEEE Transactions on Image Processing, 29, 736-746. https://doi.org/10.1109/tip.2019.2934572
  16. Jung, H.J., Lee, J.H., Yoon, S. and Kim, I.H. (2019), "Bridge inspection and condition assessment using unmanned aerial vehicles (UAVs): major challenges and solutions from a practical perspective", Smart Struct. Syst., Int. J., 24(5), 669-681. https://doi.org/10.12989/sss.2019.24.5.669
  17. Lai, Z., Alzugaray, I., Chli, M. and Chatzi, E. (2020), "Full-field structural monitoring using event cameras and physics-informed sparse identification", Mech. Syst. Signal Process., 145, 106905. https://doi.org/10.1016/j.ymssp.2020.106905
  18. Lee, K.Y. and Sim, J.Y. (2020), "Warping residual based image stitching for large parallax", 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, Washington, USA, June.
  19. Lowe, D.G. (2004), "Distinctive image features from scale-invariant keypoints", Int. J. Comput. Vis., 60(2), 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  20. Luo, J. and Oubong, G. (2009), "A comparison of SIFT, PCA-SIFT and SURF", Int. J. Image Process., 3(4), 143-152.
  21. Microsoft: Image Composite Editor (2008), Image Composite Editor - Microsoft Research, Microsoft, Redmond, USA. https://www.microsoft.com/enus/research/product/computational-photographyapplications/image-composite-editor/
  22. Moore, M., Phares, B.M., Graybeal, B., Rolander, D. and Washer, G. (2001), "Reliability of Visual Inspection for Highway Bridges", Final Rep. No. FHWA-RD-01-020, Federal Highway Administration (FHWA).
  23. Nie, L., Lin, C., Liao, K., Liu, S. and Zhao, Y. (2021), "Unsupervised deep image stitching: reconstructing stitched features to images", IEEE Transact. Image Process., 30, 6184-6197. https://doi.org/10.1109/TIP.2021.3092828
  24. OpenCV (2011), OpenCV: Images stitching, OpenCV team, USA. https://docs.opencv.org/master/d1/d46/group__stitching.html
  25. Rosten, E., Porter, R. and Drummond, T. (2010), "Faster and better: a machine learning approach to corner detection", IEEE Transact. Pattern Anal. Mach. Intell., 32(1), 105-119. https://doi.org/10.1109/TPAMI.2008.275
  26. Rousseeuw, P.J. (1984), "Least median of squares regression", J. Am. Statist. Assoc., 79(388), 871-880. https://doi.org/10.1080/01621459.1984.10477105
  27. Rublee, E., Rabaud, V., Konolige, K. and Bradski, G. (2011), "ORB: An efficient alternative to SIFT or SURF", Proceedings of 2011 International Conference on Computer Vision, Barcelona, Spain, November. https://doi.org/10.1109/ICCV.2011.6126544
  28. Schonberger, J.L. and Frahm, J. (2016), "Structure-from-motion revisited", Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, June.
  29. Spencer, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Engineering, 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  30. Strauss, A., Bien, J., Neuner, H., Harmening, C., Seywald, C., Osterreicher, M., Voit, K., Pistone, E., Spyridis, P. and Bergmeister, K. (2020), "Sensing and monitoring in tunnels testing and monitoring methods for the assessment of tunnels", Struct. Concrete, 21(4), 1356-1376. https://doi.org/10.1002/suco.201900444
  31. Sun, Y., Zhao, L., Huang, S., Yan, L. and Dissanayake, G. (2014), "L2-SIFT: SIFT feature extraction and matching for large images in large-scale aerial photogrammetry", ISPRS J. Photogram. Remote Sens., 91, 1-16. https://doi.org/10.1016/j.isprsjprs.2014.02.001
  32. Sun, L., Shang, Z., Xia, Y., Bhowmick, S. and Nagarajaiah, S. (2020), "Review of bridge structural health monitoring aided by big data and artificial intelligence: from condition assessment to damage detection", J. Struct. Eng., 146(5), 04020073. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002535
  33. Szeliski, R. (2007), Image Alignment And Stitching: A Tutorial, Now Publishers Inc., Boston, MA, USA.
  34. Torr, P.H.S. and Zisserman, A. (2000), "MLESAC: a new robust estimator with application to estimating image geometry", Comput. Vis. Image Underst., 78(1), 138-156. https://doi.org/10.1006/cviu.1999.0832
  35. VLFeat - Home (2008), VLFeat: An open and portable library of computer vision algorithms, Vedaldi A. and Fulkerson B. http://www.vlfeat.org
  36. Wu, R.T. and Jahanshahi, M.R. (2020), "Data fusion approaches for structural health monitoring and system identification: past, present, and future", Struct. Health Monitor., 19(2), 552-586. https://doi.org/10.1177/1475921718798769
  37. Xiang, T., Xia, G. and Zhang, L. (2019), "Mini-unmanned aerial vehicle-based remote sensing: techniques, applications, and prospects", IEEE Geosci. Remote Sens. Magaz., 7(3), 29-63. https://doi.org/10.1109/MGRS.2019.2918840
  38. Ye, X.W., Dong, C.Z. and Liu, T. (2016), "Image-based structural dynamic displacement measurement using different multi-object tracking algorithms", Smart Struct. Syst., Int. J., 17(6), 935-956. https://doi.org/10.12989/sss.2016.17.6.935
  39. Yeum, C.M., Choi, J. and Dyke, S.J. (2017), "Autonomous image localization for visual inspection of civil infrastructure", Smart Mater. Struct., 26(3), 035051. https://doi.org/10.1088/1361-665X/aa510e
  40. Yuan, M., Lei, T., Liu, X. and Li, S. (2019), "Fast image stitching of unmanned aerial vehicle remote sensing image based on SURF algorithm", Proceedings of the 11th International Conference on Digital Image Processing (ICDIP 2019), Guangzhou, China, May.
  41. Zhou, W., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P. (2004), "Image quality assessment: from error visibility to structural similarity", IEEE Transact. Image Process., 13(4), 600-612. https://doi.org/10.1109/TIP.2003.819861