Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Development of a structural inspection system with marking damage information at onsite based on an augmented reality technique

  • Junyeon Chung (Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology) ;
  • Kiyoung Kim (Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology) ;
  • Hoon Sohn (Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology)
  • Received : 2023.03.27
  • Accepted : 2023.05.26
  • Published : 2023.06.25
⟨ Previous Next ⟩

Abstract

Although unmanned aerial vehicles have been used to overcome the limited accessibility of human-based visual inspection, unresolved issues still remain. Onsite inspectors face difficulty finding previously detected damage locations and tracking their status onsite. For example, an inspector still marks the damage location on a target structure with chalk or drawings while comparing the current status of existing damages to their previous status, as documented onsite. In this study, an augmented-reality-based structural inspection system with onsite damage information marking was developed to enhance the convenience of inspectors. The developed system detects structural damage, creates a holographic marker with damage information on the actual physical damage, and displays the marker onsite via an augmented reality headset. Because inspectors can view a marker with damage information in real time on the display, they can easily identify where the previous damage has occurred and whether the size of the damage is increasing. The performance of the developed system was validated through a field test, demonstrating that the system can enhance convenience by accelerating the inspector's essential tasks such as detecting damages, measuring their size, manually recording their information, and locating previous damages.

Keywords

Acknowledgement

This work was supported by National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (grant numbers 2022R1C1C2008186). This study was supported by a grant (2021-MOIS32-041) from the Ministry of Interior and Safety (MOIS, Korea). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).

References

  1. Ali, R., Kang, D., Suh, G. and Cha, Y.J. (2021), "Real-time multiple damage mapping using autonomous UAV and deep faster region-based neural networks for GPS-denied structures", Autom. Constr., 130(103831). https://doi.org/10.1016/j.autcon.2021.103831
  2. Bahri, H., Krcmarik, D., Moezzi, R. and Koci, J. (2019), "Efficient use of mixed reality for BIM system using Microsoft HoloLens", IFAC-Papers Online, 52(27) 235-239. https://doi.org/10.1016/j.ifacol.2019.12.762
  3. Bolya, D., Zhou, C., Xiao, F. and Lee, Y.J. (2019), "Yolact: Realtime instance segmentation", Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9157-9166.
  4. Dai, A., Niessner, M., Zollhofer, M., Izadi, S. and Theobalt, C. (2017), "BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration", ACM Trans. Graph., 36(3), 1-18. https://doi.org/10.1145/3054739
  5. Falorca, J.F., Miraldes, J.P. and Lanzinha, J.C.G. (2021), "New trends in visual inspection of buildings and structures: study for the use of drones", Open Eng., 11(1), 734-743. https://doi.org/10.1515/eng-2021-0071
  6. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A. and Fitzgibbon, A. (2011), "KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera", Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA.
  7. Karaaslan, E., Bagci, U. and Catbas, F.N. (2019), "Artificial intelligence assisted infrastructure assessment using mixed reality systems", Transp. Res. Rec., 2673(12). https://doi.org/10.1177/0361198119839988
  8. Kim, B. and Cho, S. (2020), "Automated multiple concrete damage detection using instance segmentation deep learning model", J. Appl. Sci. 10(22), 1-17. https://doi.org/10.3390/app10228008
  9. Kim, I.H., Jeon, H., Baek, S.C., Hong, W.H. and Jung, H.J. (2018), "Application of crack identification techniques for an aging concrete bridge inspection using an un- manned aerial vehicle", Sensors, 18(6), 1-14. https://doi.org/10.3390/s18061881
  10. Li, S., Zhao, X. and Zhou, G. (2019), "Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network", Comput.Aided Civ. Infrastruct. Eng., 34(7), 616-634. https://doi.org/10.1111/mice.12433
  11. Maharjan, D., Aguero, M., Mascarenas, D., Fierro, R. and Moreu, F. (2020), "Enabling human-infrastructure interfaces for inspection using augmented reality", Struct. Health Monitor., 20(4), 147592172097701. https://doi.org/10.1177/1475921720977017
  12. Mascarenas, D.D., Ballor, J.P., McClain, O.L., Mellor, M.A., Shen, C.Y., Bleck, B., Morales, J., Yeong, L.M.R., Narushof, B., Shelton, P. and Martinez, E. (2020), "Augmented reality for next generation infrastructure inspections", Struct. Health Monitor., 20(4), 147592172095384. https://doi.org/10.1177/1475921720953846
  13. Nguyen, D.C., Nguyen, T.Q., Jin, R., Jeon, C.H. and Shim, C.S. (2021), "BIM-based mixed-reality application for bridge inspection and maintenance", Constr. Innov., 22(3), 487-503. https://doi.org/10.1108/ci-04-2021-0069
  14. Niessner, M., Zollhofer, M., Izadi, S. and Stamminger, M. (2013), "Real-time 3D reconstruction at scale using voxel hashing", ACM Trans. Graph, 32(6), 1-11. https://doi.org/10.1145/2508363.2508374
  15. Ong, S. (2021), Beginning Windows Mixed Reality Programming, (1st ed.), Apress, Berkeley, CA, USA. https://doi.org/10.1007/978-1-4842-2769-5
  16. Puz, G., Radic, J. and Tenzera, D. (2012), "Visual inspection in evaluation of bridge condition", Gradevinar, 64(9), 717-726. https://doi.org/10.14256/jce.718.2012
  17. Rublee, E., Rabaud, V., Konolige, K. and Bradski, G. (2011), "ORB: an efficient alternative to SIFT or SURF", Proceedings of 2011 International Conference on Computer Vision, Barcelona, Spain, pp. 2564-2571. https://doi.org/10.1109/ICCV.2011.6126544
  18. Sofiiuk, K., Petrov, I., Barinova, O. and Konushin, A. (2020), "fbrs: Rethinking backpropagating refinement for interactive segmentation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8623-8632.
  19. Wang, S., Zargar, S.A. and Yuan, F.G. (2021), "Augmented reality for enhanced visual inspection through knowledge-based deep learning", Struct. Health Monit., 20(1), 426-442. https://doi.org/10.1177/1475921720976986
  20. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T. and Yang, X. (2018a), "Automatic pixel-level crack detection and measurement using fully convolutional network", Comput. Aided Civ. Infrastruct. Eng., 33(12), 1090-1109. https://doi.org/10.1111/mice.12412
  21. Yang, L., Li, B., Li, W., Jiang, B. and Xiao, J. (2018b), "Semantic metric 3D reconstruction for concrete inspection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt-Lake City, UT, USA, pp. 1624-1632. https://doi.org/10.1109/CVPRW.2018.00204
  22. Zhang, C., Chang, C.C. and Jamshidi, M. (2020), "Concrete bridge surface damage detection using a single-stage detector", Comput.-Aided Civ. Infrastruct. Eng., 35(4), 389-409. https://doi.org/10.1111/mice.12500
  23. Zhang, C., Chang, C.C. and Jamshidi, M. (2021), "Simultaneous pixel-level concrete defect detection and grouping using a fully convolutional model", Struct. Health Monitor., 20(4), 2199-2215. https://doi.org/10.1177/1475921720985437