Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Deep reinforcement learning for optimal life-cycle management of deteriorating regional bridges using double-deep Q-networks

  • Xiaoming, Lei (Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University) ;
  • You, Dong (Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University)
  • Received : 2022.04.09
  • Accepted : 2022.09.05
  • Published : 2022.12.25
⟨ Previous Next ⟩

Abstract

Optimal life-cycle management is a challenging issue for deteriorating regional bridges. Due to the complexity of regional bridge structural conditions and a large number of inspection and maintenance actions, decision-makers generally choose traditional passive management strategies. They are less efficiency and cost-effectiveness. This paper suggests a deep reinforcement learning framework employing double-deep Q-networks (DDQNs) to improve the life-cycle management of deteriorating regional bridges to tackle these problems. It could produce optimal maintenance plans considering restrictions to maximize maintenance cost-effectiveness to the greatest extent possible. DDQNs method could handle the problem of the overestimation of Q-values in the Nature DQNs. This study also identifies regional bridge deterioration characteristics and the consequence of scheduled maintenance from years of inspection data. To validate the proposed method, a case study containing hundreds of bridges is used to develop optimal life-cycle management strategies. The optimization solutions recommend fewer replacement actions and prefer preventative repair actions when bridges are damaged or are expected to be damaged. By employing the optimal life-cycle regional maintenance strategies, the conditions of bridges can be controlled to a good level. Compared to the nature DQNs, DDQNs offer an optimized scheme containing fewer low-condition bridges and a more costeffective life-cycle management plan.

Keywords

Acknowledgement

The study has been supported by the Research Grant Council of Hong Kong (project no. PolyU 15219819 and 15221521). The support is gratefully acknowledged. The opinions and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the sponsoring organizations.

References

  1. Andriotis, C.P. and Papakonstantinou, K.G. (2021), "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints", Reliabil. Eng. Syst. Safety, 212, P. 107551. https://doi.org/10.1016/j.ress.2021.107551
  2. Arulkumaran, K., Deisenroth, M.P., Brundage, M. and Bharath, A.A. (2017), "Deep reinforcement learning: A brief survey", IEEE Signal Process. Magaz., 34(6), 26-38. https://doi.org/10.1109/msp.2017.2743240
  3. Bocchini, P. and Frangopol, D.M. (2011), "Probabilistic bridge network life-cycle connectivity assessment and optimization", In: Applications of Statistics and Probability in Civil Engineering, pp. 160-161.
  4. Bocchini, P., Saydam, D. and Frangopol, D.M. (2013), "Efficient, accurate, and simple Markov chain model for the life-cycle analysis of bridge groups", Struct. Safety, 40, 51-64. https://doi.org/10.1016/j.strusafe.2012.09.004
  5. Chalabi, Z. and Lorenc, T. (2013), "Using agent-based models to inform evaluation of complex interventions: Examples from the built environment", Prevent. Medic., 57(5), 434-435. https://doi.org/10.1016/j.ypmed.2013.07.013
  6. Chen, S.R., Cai, C.S. and Levitan, M. (2007), "Understand and improve dynamic performance of transportation system - a case study of Luling Bridge", Eng. Struct., 29(6), 1043-1051. https://doi.org/10.1016/j.engstruct.2006.07.019
  7. Cheng, M. and Frangopol, D.M. (2021a), "A decision-making framework for load rating planning of aging bridges using deep reinforcement learning", J. Comput. Civil Eng., 35(6), p. 04021024. https://doi.org/10.1061/(asce)cp.1943-5487.0000991
  8. Cheng, M. and Frangopol, D.M. (2021b), "Optimal load rating-based inspection planning of corroded steel girders using Markov decision process", Probabil. Eng. Mech., 66, p. 103160. https://doi.org/10.1016/j.probengmech.2021.103160
  9. Cheng, M.Y., Chiu, Y.F., Chiu, C.K., Prayogo, D., Wu, Y.W., Hsu, Z.L. and Lin, C.H. (2019), "Risk-based maintenance strategy for deteriorating bridges using a hybrid computational intelligence technique: a case study", Struct. Infrastr. Eng., 15(3), 334-350. https://doi.org/10.1080/15732479.2018.1547767
  10. Dong, Y., Frangopol, D.M. and Saydam, D. (2014), "Sustainability of highway bridge networks under seismic hazard", J. Earthq. Eng., 18(1), 41-66. https://doi.org/10.1080/13632469.2013.841600
  11. Fan, W., Chen, Y., Li, J., Sun, Y., Feng, J., Hassanin, H. and Sareh, P. (2021), "Machine learning applied to the design and inspection of reinforced concrete bridges: Resilient methods and emerging applications", Structures, 33, 3954-3963. https://doi.org/10.1016/j.istruc.2021.06.110
  12. Fiorillo, G. and Nassif, H. (2020), "Improving the conversion accuracy between bridge element conditions and NBI ratings using deep convolutional neural networks", Struct. Infrastr. Eng., 16(12), 1669-1682. https://doi.org/10.1080/15732479.2020.1725065
  13. Frangopol, D.M. and Bocchini, P. (2012), "Bridge network performance, maintenance and optimisation under uncertainty: accomplishments and challenges", Struct. Infrastr. Eng., 8(4), 341-356. https://doi.org/10.1080/15732479.2011.563089
  14. Frangopol, D.M., Dong, Y. and Sabatino, S. (2017), "Bridge life-cycle performance and cost: analysis, prediction, optimisation and decision-making", Struct. Infrastr. Eng., 13(10), 1239-1257. https://doi.org/10.1080/15732479.2016.1267772
  15. Ge, B. and Kim, S. (2021a), "Determination of appropriate updating parameters for effective life-cycle management of deteriorating structures under uncertainty", Struct. Infrastr. Eng., 17(9), 1284-1298. https://doi.org/10.1080/15732479.2020.1809466
  16. Ge, B. and Kim, S. (2021b), "Probabilistic service life prediction updating with inspection information for RC structures subjected to coupled corrosion and fatigue", Eng. Struct., 238, p. 112260. https://doi.org/10.1016/j.engstruct.2021.112260
  17. Guan, X., Xiang, Z., Bao, Y. and Li, H. (2022), "Structural dominant failure modes searching method based on deep reinforcement learning", Reliabil. Eng. Syst. Safety, 219, p. 108258. https://doi.org/10.1016/j.ress.2021.108258
  18. Hadjidemetriou, G.M., Herrera, M. and Parlikad, A.K. (2022), "Condition and criticality-based predictive maintenance prioritisation for networks of bridges", Struct. Infrastr. Eng., 18(8), 1207-1221. https://doi.org/10.1080/15732479.2021.1897146
  19. Huynh, T.C., Nguyen, T.T., Kim, J.T., Ta, Q.B., Ho, D.D. and Phan, T.T.V. (2021), "Deep learning-based functional assessment of piezoelectric-based smart interface under various degradations", Smart Struct. Syst., Int. J., 28(1), 69-87. https://doi.org/10.12989/sss.2021.28.1.069
  20. Jung, H.J., Lee, J.H., Yoon, S.S. and Kim, I.H. (2019), "Bridge Inspection and condition assessment using Unmanned Aerial Vehicles (UAVs): Major challenges and solutions from a practical perspective", Smart Struct. Syst., Int. J., 24(5), 669-681. https://doi.org/10.12989/sss.2019.24.5.669
  21. Kabir, G., Sadiq, R. and Tesfamariam, S. (2014), "A review of multi-criteria decision-making methods for infrastructure management", Struct. Infrastr. Eng., 10(9), 1176-1210. https://doi.org/10.1080/15732479.2013.795978
  22. Law, S.S. and Li, J. (2010), "Updating the reliability of a concrete bridge structure based on condition assessment with uncertainties", Eng. Struct., 32(1), 286-296. https://doi.org/10.1016/j.engstruct.2009.09.015
  23. Lee, D.H. and Koh, B.H. (2021), "An image-based deep learning network technique for structural health monitoring", Smart Struct. Syst., Int. J., 28(6), 799-810. https://doi.org/10.12989/sss.2021.28.6.799
  24. Lee, J.H., Yoon, S., Kim, B., Gwon, G.H., Kim, I.H. and Jung, H.J. (2021), "A new image-quality evaluating and enhancing methodology for bridge inspection using an unmanned aerial vehicle", Smart Struct. Syst., Int. J., 27(2), 209-226. https://doi.org/10.12989/sss.2021.27.2.209
  25. Lei, X., Sun, L. and Xia, Y. (2020), "Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks", Struct. Health Monitor., 20(4), 2069-2087. https://doi.org/10.1177/1475921720959226
  26. Lei, X., Sun, L. and Xia, Y. (2021a), "Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks", Struct. Health Monitor. - Int. J., 20(4), 2069-2087. https://doi.org/10.1177/1475921720959226
  27. Lei, X., Sun, L. and Xia, Y. (2021b), "Seismic fragility assessment and maintenance management on regional bridges using bayesian multi-parameter estimation", Bull. Earthq. Eng., 19(15), 6693-6717. https://doi.org/10.1007/s10518-021-01072-6
  28. Lei, X., Xia, Y., Dong, Y. and Sun, L. (2022a), "Multi-level timevariant vulnerability assessment of deteriorating bridge networks with structural condition records", Eng. Struct., 266, p. 114581. https://doi.org/10.1016/j.engstruct.2022.114581
  29. Lei, X., Xia, Y., Komarizadehasl, S. and Sun, L. (2022b), "Condition level deteriorations modeling of RC beam bridges with U-Net convolutional neural networks", Structures, 42, 333-342. http://https://doi.org/10.1016/j.istruc.2022.06.013
  30. Li, S., Ou, J., Wang, J., Gao, X. and Yang, C. (2019), "Level 2 safety evaluation of concrete-filled steel tubular arch bridges incorporating structural health monitoring and inspection information based on China bridge standards", Struct. Control Health Monitor., 26(3), p. e2303. https://doi.org/10.1002/stc.2303
  31. Liu, H. and Zhang, Y. (2020), "Bridge condition rating data modeling using deep learning algorithm", Struct. Infrastr. Eng., 16(10), 1447-1460. https://doi.org/10.1080/15732479.2020.1712610
  32. Liu, K., Zhai, C. and Dong, Y. (2021), "Optimal restoration schedules of transportation network considering resilience", Struct. Infrastr. Eng., 17(8), 1141-1154. https://doi.org/10.1080/15732479.2020.1801764
  33. Lydon, D., Taylor, S.E., Lydon, M., del Rincon, J.M. and Hester, D. (2019), "Development and testing of a composite system for bridge health monitoring utilising computer vision and deep learning", Smart Struct. Syst., Int. J., 24(6), 723-732. https://doi.org/10.12989/sss.2019.24.6.723
  34. Memarzadeh, M. and Pozzi, M. (2019), "Model-free reinforcement learning with model-based safe exploration: Optimizing adaptive recovery process of infrastructure systems", Struct. Safety, 80, 46-55. https://doi.org/10.1016/j.strusafe.2019.04.003
  35. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G. and Petersen, S. (2015), "Human-level control through deep reinforcement learning", Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236
  36. Mondal, T.G. and Jahanshahi, M.R. (2020), "Autonomous vision-based damage chronology for spatiotemporal condition assessment of civil infrastructure using unmanned aerial vehicle", Smart Struct. Syst., Int. J., 25(6), 733-749. https://doi.org/10.12989/sss.2020.25.6.733
  37. Oh, B.K., Glisic, B. and Park, H.S. (2021), "Convolutional neural network-based damage detection method for building structures", Smart Struct. Syst., Int. J., 27(6), 903-916. https://doi.org/10.12989/sss.2021.27.6.903
  38. Ti, Z., Deng, X.W. and Yang, H. (2020), "Wake modeling of wind turbines using machine learning", Appl. Energy, 257, p. 114025. https://doi.org/10.1016/j.apenergy.2019.114025
  39. Ti, Z., Deng, X.W. and Zhang, M. (2021), "Artificial Neural Networks based wake model for power prediction of wind farm", Renewable Energy, 172, 618-631. https://doi.org/10.1016/j.renene.2021.03.030
  40. Van Hasselt, H., Guez, A. and Silver, D. (2016), "Deep reinforcement learning with double q-learning", Proceedings of the 13th AAAI Conference on Artificial Intelligence, Vol. 30, No. 1, pp. 2094-2100, Phoenix. AZ, USA, February. https://doi.org/10.1609/aaai.v30i1.10295
  41. Wang, Z., Dong, Y. and Jin, W. (2021), "Life-cycle cost analysis of deteriorating civil infrastructures incorporating social sustainability", J. Infrastr. Syst., 27(3), p. 04021013. https://doi.org/10.1061/(asce)is.1943-555x.0000607
  42. Wei, S., Bao, Y. and Li, H. (2020), "Optimal policy for structure maintenance: A deep reinforcement learning framework", Struct. Safety, 83, p. 101906. https://doi.org/10.1016/j.strusafe.2019.101906
  43. Xia, H.W., Ni, Y.Q., Wong, K.Y. and Ko, J.M. (2012), "Reliability-based condition assessment of in-service bridges using mixture distribution models", Comput. Struct., 106, 204-213. https://doi.org/10.1016/j.compstruc.2012.05.003
  44. Xia, Y., Lei, X., Wang, P., Liu, G. and Sun, L. (2020), "Long-term performance monitoring and assessment of concrete beam bridges using neutral axis indicator", Struct. Control Health Monitor., 27(12), p. e2637. https://doi.org/10.1002/stc.2637
  45. Xia, Y., Lei, X., Wang, P. and Sun, L. (2021), "Artificial intelligence based structural assessment for regional short-and medium-span concrete beam bridges with inspection information", Remote Sens., 13(18), p. 3687. https://doi.org/10.3390/rs13183687
  46. Xia, Y., Lei, X., Wang, P. and Sun, L. (2022), "A data-driven approach for regional bridge condition assessment using inspection reports", Struct. Control Health Monitor., 29(4), p. e2915. https://doi.org/10.1002/stc.2915
  47. Yang, D.Y. (2022), "Adaptive Risk-Based Life-Cycle Management for Large-Scale Structures Using Deep Reinforcement Learning and Surrogate Modeling", J. Eng. Mech., 148(1), p. 04021126. https://doi.org/10.1061/(asce)em.1943-7889.0002028
  48. Yang, X.M., Yi, T.H., Qu, C.X., Li, H.N. and Liu, H. (2020), "Continuous tracking of bridge modal parameters based on subspace correlations", Struct. Control Health Monitor., 27(10), p. e2615. https://doi.org/10.1002/stc.2615
  49. Yang, S., Deng, X., Ti, Z., Yan, B. and Yang, Q. (2022), "Cooperative yaw control of wind farm using a double-layer machine learning framework", Renew. Energy, 193, 519-537. https://doi.org/10.1016/j.renene.2022.04.104
  50. Ye, X.W., Jin, T. and Yun, C.B. (2019), "A review on deep learning-based structural health monitoring of civil infrastructures", Smart Struct. Syst., Int. J., 24(5), 567-585. https://doi.org/10.12989/sss.2019.24.5.567
  51. Yi, T.H., Zhang, J., Qu, C.X. and Li, H.N. (2021), "Damage detection for decks of concrete girder bridges using the frequency obtained from an actively excited vehicle", Smart Struct. Syst., Int. J., 27(1), 101-114. https://doi.org/10.12989/sss.2021.27.1.101
  52. Zhang, N. and Si, W. (2020), "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks", Reliabil. Eng. Syst. Safety, 203, p. 107094. https://doi.org/10.1016/j.ress.2020.107094
  53. Zhang, Y., Kim, C.W., Zhang, L., Bai, Y., Yang, H., Xu, X. and Zhang, Z. (2020), "Long term structural health monitoring for old deteriorated bridges: a copula-ARMA approach", Smart Struct. Syst., Int. J., 25(3), 285-299. https://doi.org/10.12989/sss.2020.25.3.285
  54. Zheng, X., Yang, D.H., Yi, T.H. and Li, H.N. (2021), "Bridge influence surface identification method considering the spatial effect of vehicle load", Struct. Control Health Monitor., 28(8), p. e2769. https://doi.org/10.1002/stc.2769
  55. Zhu, J. and Wang, Y. (2021), "Feature Selection and Deep Learning for Deterioration Prediction of the Bridges", J. Perform. Constr. Facil., 35(6), p. 04021078. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001653