IEMEK Journal of Embedded Systems and Applications (대한임베디드공학회논문지)

Institute of Embedded Engineering of Korea (대한임베디드공학회)
권호 표지
DOI QR코드

DOI QR Code

Dueling DQN-based Routing for Dynamic LEO Satellite Networks

동적 저궤도 위성 네트워크를 위한 Dueling DQN 기반 라우팅 기법

  • Dohyung Kim (Department of IT Convergence Engineering, Kumoh National Institute of Technology ) ;
  • Sanghyeon Lee (Research Institute of Manufacturing Technology, Kumoh National Institute of Technology ) ;
  • Heoncheol Lee (Department of IT Convergence Engineering, Kumoh National Institute of Technology ) ;
  • Dongshik Won (TelePIX Co., Ltd )
  • Received : 2023.04.20
  • Accepted : 2023.06.14
  • Published : 2023.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

This paper deals with a routing algorithm which can find the best communication route to a desired point considering disconnected links in the LEO (low earth orbit) satellite networks. If the LEO satellite networks are dynamic, the number and distribution of the disconnected links are varying, which makes the routing problem challenging. To solve the problem, in this paper, we propose a routing method based on Dueling DQN which is one of the reinforcement learning algorithms. The proposed method was successfully conducted and verified by showing improved performance by reducing convergence times and converging more stably compared to other existing reinforcement learning-based routing algorithms.

Keywords

Acknowledgement

이 연구는 2022년 정부 (방위사업청)의 재원으로 국방과학연구소의 지원을 받아 수행된 연구임 (UI220033VD).

References

  1. B. S. Roh, M. H. Han, D. W. Kum, K. S. Jeon, "A Study on the Reinforcement Learning Routing for LEO Satellite Network", Proceedings of the Korean Institute of Communication Sciences Conference, pp.537-538, 2022. 
  2. J. H. Lee, Y. C. Ko, "A Study on the Low-earth Orbit Satellite Based Non-terrestrial Network Systems Via Deep-reinforcement Learning", Proceedings of the Korean Institute of Communication Sciences Conference, pp.1306-1307, 2021. 
  3. X. Wang, Z. Dai, Z. Xu. "LEO Satellite Network Routing Algorithm Based on Reinforcement Learning." In 2021 IEEE 4th International Conference on Electronics Technology (ICET), pp. 1105-1109. IEEE, 2021. 
  4. P. Zuo, C. Wang, Z. Yao, S. Hou, H. Jiang. "An Intelligent Routing Algorithm for Leo Satellites Based on Deep Reinforcement Learning." In 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), pp. 1-5. IEEE, 2021. 
  5. A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, G. Chowdhary. "Robust Deep Reinforcement Learning with Adversarial Attacks." arXiv preprint arXiv:1712.03632 (2017). 
  6. A. Cigliano, F. Zampognaro. "A Machine Learning Approach for Routing in Satellite Mega-Constellations." In 2020 International Symposium on Advanced Electrical and Communication Technologies (ISAECT), pp. 1-6. IEEE, 2020. 
  7. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. "Playing Atari with Deep Reinforcement Learning." arXiv preprint arXiv:1312.5602 (2013). 
  8. R. S. Sutton, A. G. Barto, "Reinforcement Learning: An Introduction." MIT press, 2018. 
  9. Y. Burda, H. Edwards, A. Storkey, O. Klimov "Exploration by Random Network Distillation." arXiv preprint arXiv:1810.12894 (2018). 
  10. W. Zhao, J. P. Queralta, T. Westerlund. "Sim-to-real Transfer in Deep Reinforcement Learning for Robotics: a Survey." In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 737-744. IEEE, 2020. 
  11. Z. Wang, T. Schaul, M. Hessel, H. V. Hasselt, M. Lanctot, Na. de Freitas, "Dueling Network Architectures for Deep Reinforcement Learning." In International Conference on Machine Learning, pp. 1995-2003. PMLR, 2016. 
  12. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, D. Hassabis, "Human-level Control Through Deep Reinforcement Learning." Nature 518, No. 7540, pp. 529-533, 2015.  https://doi.org/10.1038/nature14236
  13. J. H. Baek, H. T. Oh, S. J. Lee, S. H. Kim. "A Study about Application of Indoor Autonomous Driving for Obstacle Avoidance Using Atari Deep Q Network Model." In Proceedings of the Korea Information Processing Society Conference, pp. 715-718. Korea Information Processing Society, 2018. 
  14. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra. "Continuous Control with Deep Reinforcement Learning." arXiv preprint arXiv:1509.02971 (2015). 
  15. T. Schaul, J. Quan, I. Antonoglou, D. Silver. "Prioritized Experience Replay." arXiv preprint arXiv:1511.05952 (2015). 
  16. V. Konda, J. Tsitsiklis. "Actor-critic Algorithms." Advances in Neural Information Processing Systems 12 (1999). 
  17. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. "Proximal Policy Optimization Algorithms." arXiv preprint arXiv:1707.06347 (2017).