Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Corroded and loosened bolt detection of steel bolted joints based on improved you only look once network and line segment detector

  • Youhao Ni (Key Laboratory of C&PC Structures of Ministry of Education, Southeast University) ;
  • Jianxiao Mao (Key Laboratory of C&PC Structures of Ministry of Education, Southeast University) ;
  • Hao Wang (Key Laboratory of C&PC Structures of Ministry of Education, Southeast University) ;
  • Yuguang Fu (School of Civil and Environmental Engineering, Nanyang Technological University) ;
  • Zhuo Xi (Key Laboratory of C&PC Structures of Ministry of Education, Southeast University)
  • Received : 2022.12.04
  • Accepted : 2023.06.30
  • Published : 2023.07.25
⟨ Previous Next ⟩

Abstract

Steel bolted joint is an important part of steel structure, and its damage directly affects the bearing capacity and durability of steel structure. Currently, the existing research mainly focuses on the identification of corroded bolts and corroded bolts respectively, and there are few studies on multiple states. A detection framework of corroded and loosened bolts is proposed in this study, and the innovations can be summarized as follows: (i) Vision Transformer (ViT) is introduced to replace the third and fourth C3 module of you-only-look-once version 5s (YOLOv5s) algorithm, which increases the attention weights of feature channels and the feature extraction capability. (ii) Three states of the steel bolts are considered, including corroded bolt, bolt missing and clean bolt. (iii) Line segment detector (LSD) is introduced for bolt rotation angle calculation, which realizes bolt looseness detection. The improved YOLOv5s model was validated on the dataset, and the mean average precision (mAP) was increased from 0.902 to 0.952. In terms of a lab-scale joint, the performance of the LSD algorithm and the Hough transform was compared from different perspective angles. The error value of bolt loosening angle of the LSD algorithm is controlled within 1.09%, less than 8.91% of the Hough transform. Furthermore, the proposed framework was applied to fullscale joints of a steel bridge in China. Synthetic images of loosened bolts were successfully identified and the multiple states were well detected. Therefore, the proposed framework can be alternative of monitoring steel bolted joints for management department.

Keywords

Acknowledgement

The authors greatly acknowledge the financial support from the National Natural Science Foundation of China (51978155, 52108274) and Postgraduate Research & Practice Innovation Program of Jiangsu Province (grant number: SJCX21_0056). The authors also would like to thank Postdoctoral Fellow Yiming Zhang of The Hong Kong Polytechnic University, doctoral candidate Hui Gao, doctoral candidate Ruijun Liang and doctoral candidate Zhijie Yuan of Southeast University for paper writing and computing. Finally, contributions by the anonymous reviewers are also highly appreciated.

References

  1. Atha, D.J. and Jahanshahi, M.R. (2018), "Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection", Struct. Health Monitor., 17(5), 1110-1128. https://doi.org/10.1177/1475921717737051
  2. Baevski, A. and Auli, M. (2018), "Adaptive input representations for neural language modeling", arXiv preprint arXiv:1809.10853. https://doi.org/10.48550/arXiv.1809.10853
  3. Blachowski, B., Swiercz, A. and Pnevmatikos, N. (2015), "Experimental verification of damage location techniques for frame structures assembled using bolted connections", Proceedings of the 5th International Conference on Computational Methods in Structural Dynamics and Earthquake Engineering, pp. 05-25.
  4. Bochkovskiy, A., Wang, C.Y. and Liao, H.Y.M. (2020), "Yolov4: Optimal speed and accuracy of object detection", arXiv preprint arXiv:2004.10934. https://doi.org/10.48550/arXiv.2004.10934
  5. Caesar, H., Uijlings, J. and Ferrari, V. (2018), "Coco-stuff: Thing and stuff classes in context", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209-1218.
  6. Cha, Y.J., You, K. and Choi, W. (2016), "Vision-based detection of loosened bolts using the Hough transform and support vector machines", Automat. Constr., 71, 181-188. https://doi.org/10.1016/j.autcon.2016.06.008
  7. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S. and Buyukozturk, O. (2018), "Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types", Comput-Aided. Civil Infrastr., Eng., 33(9), 731-747. https://doi.org/10.1111/mice.12334
  8. Chaki, S. and Bourse, G. (2009), "Stress level measurement in prestressed steel strands using acoustoelastic effect", Exp. Mech., 49(5), 673-681. https://doi.org/10.1007/s11340-008-9174-9
  9. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2018), "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805. https://doi.org/10.48550/arXiv.1810.04805
  10. Ding, L. and Goshtasby, A. (2001), "On the Canny edge detector", Pattern Recognit., 34(3), 721-725. https://doi.org/10.1016/S0031-3203(00)00023-6
  11. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G. and Gelly, S. (2020), "An image is worth 16x16 words: Transformers for image recognition at scale", arXiv preprint arXiv:2010.11929. https://doi.org/10.48550/arXiv.2010.11929
  12. Forkan, A.R.M., Kang, Y.B., Jayaraman, P.P., Liao, K., Kaul, R., Morgan, G., Ranjan, R. and Sinha, S. (2022), "CorrDetector: A framework for structural corrosion detection from drone images using ensemble deep learning", Expert Syst. Appl., 193, 116461. https://doi.org/10.1016/j.eswa.2021.116461
  13. Ge, Z., Liu, S., Wang, F., Li, Z. and Sun, J. (2021), "Yolox: Exceeding yolo series in 2021", arXiv preprint arXiv:2107.08430. https://doi.org/10.48550/arXiv.2107.08430
  14. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J. and Chen, T. (2018), "Recent advances in convolutional neural networks", Pattern Recogn., 77, 354-377. https://doi.org/10.1016/j.patcog.2017.10.013
  15. Hoskere, V., Narazaki, Y., Hoang, T. and Spencer Jr, B. (2018), "Vision-based structural inspection using multiscale deep convolutional neural networks", arXiv preprint arXiv:1805.01055. https://doi.org/10.48550/arXiv.1805.01055
  16. Huynh, T.C. (2021), "Vision-based autonomous bolt-looseness detection method for splice connections: Design, lab-scale evaluation, and field application", Automat. Constr., 124, 103591. https://doi.org/10.1016/j.autcon.2021.103591
  17. Huynh, T.C., Park, J.H., Jung, H.-J. and Kim, J.T. (2019), "Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing", Automat. Constr., 105, 102844. https://doi.org/10.1016/j.autcon.2019.102844
  18. Jiang, Y., Zhang, M. and Lee, C.-H. (2003), "A study of early stage self-loosening of bolted joints", J. Mech. Des., 125(3), 518-526. https://doi.org/10.1115/1.1586936
  19. Junker, G.H. (1969), "New criteria for self-loosening of fasteners under vibration", Sae Transact., 78, 314-335. https://www.jstor.org/stable/44563013
  20. Mezirow, J. (1978), "Perspective transformation", Adult education, 28(2), 100-110. https://doi.org/10.1177/074171367802800202
  21. Muthalagu, R., Bolimera, A. and Kalaichelvi, V. (2020), "Lane detection technique based on perspective transformation and histogram analysis for self-driving cars", Comput. Electr. Eng., 85, 106653. https://doi.org/10.1016/j.compeleceng.2020.106653
  22. Ni, Y.H., Mao, J.X., Wang, H., Xi, Z. and Xu, Y.F. (2023), "Toward high-precision crack detection in concrete bridges using deep learning", J. Perform. Constr. Facil., 37(3), 04023017. https://doi.org/10.1061/JPCFEV.CFENG-4275
  23. Nikravesh, S.M.Y. and Goudarzi, M. (2017), "A review paper on looseness detection methods in bolted structures", Lat. Am. J. Solids Struct., 14, 2153-2176. https://doi.org/10.1590/1679-78254231
  24. Pan, X. and Yang, T. (2021), "Image-based monitoring of bolt loosening through deep-learning-based integrated detection and tracking", Comput-Aided. Civil Infrastr. Eng., 37(10), 1207-1222. https://doi.org/10.1111/mice.12797
  25. Park, G., Sohn, H., Farrar, C.R. and Inman, D.J. (2003), "Overview of piezoelectric impedance-based health monitoring and path forward", Shock Vib. Digest, 35(6), 451-464. https://doi.org/10.48550/arXiv.1805.01055
  26. Pham, H.C., Ta, Q.B., Kim, J.T., Ho, D.D., Tran, X.L. and Huynh, T.C. (2020), "Bolt-loosening monitoring framework using an image-based deep learning and graphical model", Sensors, 20(12), 3382. https://doi.org/ 10.3390/s20123382
  27. Pidaparti, R.M. (2007), "Structural corrosion health assessment using computational intelligence methods", Struct. Health Monitor., 6(3), 245-259. https://doi.org/10.1177/1475921707081975
  28. Pukelsheim, F. (1994), "The three sigma rule", Am. Stat., 48(2), 88-91. https://doi.org/10.1080/00031305.1994.10476030
  29. Redmon, J. and Farhadi, A. (2017), "YOLO9000: better, faster, stronger", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263-7271.
  30. Redmon, J. and Farhadi, A. (2018), "Yolov3: An incremental improvement", arXiv preprint arXiv:1804.02767. https://doi.org/10.48550/arXiv.1804.02767
  31. Redmon, J., Divvala, S., Girshick, R. and Farhadi, A. (2016), "You only look once: Unified, real-time object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779-788.
  32. Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Trans. Pattern Anal. Mach. Intell., 39(6), 1137-1149. https://doi.org/ 10.1109/TPAMI.2016.2577031
  33. Spencer Jr, B.F., Hoskere, V. and Narazaki, Y. (2019), "Advances in computer vision-based civil infrastructure inspection and monitoring", Eng., 5(2), 199-222. https://doi.org/10.1016/j.eng.2018.11.030
  34. Sun, L., Shang, Z., Xia, Y., Bhowmick, S. and Nagarajaiah, S. (2020), "Review of bridge structural health monitoring aided by big data and artificial intelligence: From condition assessment to damage detection", J. Struct. Eng., 146(5), 04020073. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002535
  35. Ta, Q.B. and Kim, J.T. (2020), "Monitoring of corroded and loosened bolts in steel structures via deep learning and Hough transforms", Sensors, 20(23), 6888. https://doi.org/10.3390/s20236888
  36. Ta, Q.-B., Huynh, T.C., Pham, Q.Q. and Kim, J.T. (2022), "Corroded bolt identification using mask region-based deep learning trained on synthesized data", Sensors, 22(9), 3340. https://doi.org/10.3390/s22093340
  37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017), "Attention is all you need", Adv. Neural Inform. Process., 30.
  38. Von Gioi, R.G., Jakubowicz, J., Morel, J.M. and Randall, G. (2008), "LSD: A fast line segment detector with a false detection control", IEEE Trans. Pattern Anal. Mach. Intell., 32(4), 722-732. https://doi.org/10.1109/TPAMI.2008.300
  39. Wang, T., Song, G., Wang, Z. and Li, Y. (2013a), "Proof-of-concept study of monitoring bolt connection status using a piezoelectric based active sensing method", Smart Mater. Struct., 22(8), 087001. https://doi.org/10.1088/0964-1726/22/8/087001
  40. Wang, T., Song, G., Liu, S., Li, Y. and Xiao, H. (2013b), "Review of bolted connection monitoring", Int. J. Distrib. Sens. Netw., 9(12), 871213. https://doi.org/10.1155/2013/871213
  41. Wang, C., Wang, N., Ho, S.C., Chen, X. and Song, G. (2019), "Design of a new vision-based method for the bolts looseness detection in flange connections", IEEE Trans. Ind. Electron., 67(2), 1366-1375. https://doi.org/10.1109/TIE.2019.2899555
  42. Weiss, K., Khoshgoftaar, T.M. and Wang, D. (2016), "A survey of transfer learning", J. Big Data, 3(1), 1-40. https://doi.org/10.1186/s40537-016-0043-6
  43. Xu, J., Gui, C. and Han, Q. (2020), "Recognition of rust grade and rust ratio of steel structures based on ensembled convolutional neural network", Comput-Aided. Civil Infrastr. Eng., 35(10), 1160-1174. https://doi.org/10.1111/mice.12563
  44. Yang, J. and Chang, F.K. (2006), "Detection of bolt loosening in C-C composite thermal protection panels: I. Diagnostic principle", Smart Mater. Struct., 15(2), 581. https://doi.org/10.1088/0964-1726/15/2/041
  45. Ye, X.W., Dong, C.Z. and Liu, T. (2016), "A review of machine vision-based structural health monitoring: methodologies and applications", J. Sensors, 2016. https://doi.org/10.1155/2016/7103039
  46. Zhang, Y., Sun, X., Loh, K.J., Su, W., Xue, Z. and Zhao, X. (2020), "Autonomous bolt loosening detection using deep learning", Struct. Health Monitor., 19(1), 105-122. https://doi.org/10.1177/1475921719837509