International conference on construction engineering and project management (국제학술발표논문집)

  • 2024.07a
  • /
  • Pages.799-806
  • /
  • 2024
  • /
  • 2508-9048(eISSN)
Korea Institute of Construction Engineering and Management (한국건설관리학회)
권호 표지
DOI QR코드

DOI QR Code

Q-learning for tunnel excavation schedule

  • Shuhan YANG (Department of Architecture and Civil Engineering, City University of Hong Kong) ;
  • Ke DAI (Department of Architecture and Civil Engineering, City University of Hong Kong) ;
  • Zhihao REN (Department of Architecture and Civil Engineering, City University of Hong Kong) ;
  • Jung In KIM (Department of Architecture and Civil Engineering, City University of Hong Kong) ;
  • Bin XUE (School of Public Policy and Administration, Chongqing University) ;
  • Dan WANG (School of Public Policy and Administration, Chongqing University) ;
  • Wooyong JUNG (Department of Nuclear Power Plant Engineering, KEPCO International Nuclear Graduate School)
  • Published : 2024.07.29
Download PDF
⟨ Previous Next ⟩

Abstract

Construction planners for hard rock tunnel projects often encounter practical challenges caused by inherent uncertainties in ground conditions and resource constraints. Therefore, planners cannot rapidly generate optimal excavation schedules for the shortest project durations with a given equipment fleet by considering the uncertainties in ground conditions. Although some schedule optimization methods exist, they are not tailored for resource-constrained hard rock tunnel projects. To overcome these limitations, the authors specified a formal Q-learning-based schedule optimization methodology for resource-constrained hard rock tunnel projects. States are defined according to the locations of tunnel faces under excavation. Actions consist of multiple and comprehensive heuristic-based rules, which are efficient methods for resource allocation. Rewards are the time intervals required between current states and next states. After that, the methodology is validated using a case study. The generated Q tables indicate (1) best actions under different states and (2) the shortest remaining durations when the project starts from specific (state, action) pairs. The results demonstrate that the optimal schedules can be obtained by applying the proposed methodology. Furthermore, it is beneficial for planners to rapidly assign optimal rules for each state under one ground condition scenario. The results further show the potential to consider the uncertainties in ground conditions using the information of possible ground condition scenarios provided.

Keywords

Acknowledgement

The support of the General Research Fund of the University Grants Committee of Hong Kong (#21212019) and City University of Hong Kong (#9239027) is gratefully acknowledged.

References

  1. C. Paraskevopoulou and A. Benardos, "Assessing the construction cost of Greek transportation tunnel projects.," Tunnelling and Underground Space Technology. vol. 38, pp. 497-505, 2013. 
  2. I. Kurtulus and E.W. Davis, "Multi-Project Scheduling: Categorization of Heuristic Rules Performance.," Management Science. vol. 28, no. 2, pp. 161-172, 1982. 
  3. S. Tsubakitani and R.F. Deckro, "A heuristic for multi-project scheduling with limited resources in the housing industry.," European Journal of Operational Research. vol. 49, no. 1, pp. 80-91, 1990. 
  4. D.M. Tsai and H.N. Chiu, "Two heuristics for scheduling multiple projects with resource constraints.," Construction Management and Economics. vol. 14, no. 4, pp. 325-340, 1996. 
  5. Y. Wang, Z. He, L.P. Kerkhove, and M. Vanhoucke, "On the performance of priority rules for the stochastic resource constrained multi-project scheduling problem.," Computers and Industrial Engineering. vol. 114, no. October, pp. 223-234, 2017. 
  6. H.J. Chen, G. Ding, J. Zhang, and S. Qin, "Research on priority rules for the stochastic resource constrained multi-project scheduling problem with new project arrival.," Computers and Industrial Engineering. vol. 137, no. August, p. 106060, 2019. 
  7. P.H. Chen and S.M. Shahandashti, "Hybrid of genetic algorithm and simulated annealing for multiple project scheduling with multiple resource constraints.," Automation in Construction. vol. 18, no. 4, pp. 434-443, 2009. 
  8. B. Said, "EARTHWORK ALLOCATIONS WITH LINEAR UNIT COSTS.," vol. 114, no. 4, pp. 641-655, 1989. 
  9. M. Marzouk and O. Moselhi, "Multiobjective Optimization of Earthmoving Operations.," Journal of Construction Engineering and Management. vol. 130, no. 1, pp. 105-113, 2004. 
  10. K. El-Rayes and A. Kandil, "Time-Cost-Quality Trade-Off Analysis for Highway Construction.," Journal of Construction Engineering and Management. vol. 131, no. 4, pp. 477-486, 2005. 
  11. P.G. Ipsilandis, "Multiobjective Linear Programming Model for Scheduling Linear Repetitive Projects.," Journal of Construction Engineering and Management. vol. 133, no. 6, pp. 417-424, 2007. 
  12. J.I. Kim, M. Fischer, and C. Kam, "Generation and evaluation of excavation schedules for hard rock tunnels in preconstruction and construction.," Automation in Construction. vol. 96, no. October, pp. 378-397, 2018. 
  13. A.K.W. Jayawardane and F.C. Harris, "FURTHER DEVELOPMENT OF INTEGER PROGRAMMING IN EARTHWORK OPTIMIZATION.," vol. 116, no. 1, pp. 18-34, 1990. 
  14. N.N. Eldin and A.B. Senouci, "Scheduling and control of linear projects.," Canadian journal of civil engineering. vol. 21, no. 2, pp. 219-230, 1994. 
  15. A. Myat, M. Paing, and N.L. Thein, "OPTIMIZING RESOURCE UTILIZATION FOR REPETITIVE CONSTRUCTION PROJECTS.," vol. 4, no. 3, pp. 85-99, 2012. 
  16. T. Hegazy, "Optimization of construction time-cost trade-off analysis using genetic algorithms.," Canadian Journal of Civil Engineering. vol. 26, no. 6, pp. 685-697, 1999. 
  17. A. Alshibani and O. Moselhi, "Fleet selection for earthmoving projects using optimization-based simulation.," Canadian Journal of Civil Engineering. vol. 39, no. 6, pp. 619-630, 2012. 
  18. P. Ghoddousi, E. Eshtehardian, S. Jooybanpour, and A. Javanmardi, "Multi-mode resource-constrained discrete time-cost-resource optimization in project scheduling using non-dominated sorting genetic algorithm.," Automation in Construction. vol. 30, pp. 216-227, 2013. 
  19. L.D. Long and A. Ohsato, "Fuzzy critical chain method for project scheduling under resource constraints and uncertainty.," International Journal of Project Management. vol. 26, no. 6, pp. 688-698, 2008. 
  20. M. Lu, H.C. Lam, and F. Dai, "Resource-constrained critical path analysis based on discrete event simulation and particle swarm optimization.," Automation in Construction. vol. 17, no. 6, pp. 670-681, 2008. 
  21. N.S. Kedir, S. Somi, A.R. Fayek, and P.H.D. Nguyen, "Hybridization of reinforcement learning and agent-based modeling to optimize construction planning and scheduling.," Automation in Construction. vol. 142, no. July, p. 104498, 2022. 
  22. W. Genders and S. Razavi, "Asynchronous n-step Q-learning adaptive traffic signal control.," Journal of Intelligent Transportation Systems: Technology, Planning, and Operations. vol. 23, no. 4, pp. 319-331, 2019. 
  23. G.H. Erharter, T.F. Hansen, Z. Liu, and T. Marcher, "Reinforcement learning based process optimization and strategy development in conventional tunneling.," Automation in Construction. vol. 127, no. December 2020, p. 103701, 2021. 
  24. Z. Zhang, A. Chong, Y. Pan, C. Zhang, and K.P. Lam, "Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning.," Energy and Buildings. vol. 199, pp. 472-490, 2019. 
  25. H. Berlink, N. Kagan, and A.H. Reali Costa, "Intelligent Decision-Making for Smart Home Energy Management.," Journal of Intelligent and Robotic Systems: Theory and Applications. vol. 80, pp. 331-354, 2015. 
  26. R.S. Sutton and A.G. Barto, "Reinforcement Learning: An Introduction.," MIT press, 2018.