International Journal of Fuzzy Logic and Intelligent Systems

Korean Institute of Intelligent Systems (한국지능시스템학회)
권호 표지
DOI QR코드

DOI QR Code

A Reinforcement Learning with CMAC

  • Kwon, Sung-Gyu (Faculty of Mechanical and Automotive Engineering Keimyung University)
  • Published : 2006.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

To implement a generalization of value functions in Adaptive Search Element (ASE)-reinforcement learning, CMAC (Cerebellar Model Articulation Controller) is integrated into ASE controller. ASE-reinforcement learning scheme is briefly studied to discuss how CMAC is integrated into ASE controller. Neighbourhood Sequential Training for CMAC is utilized to establish the look-up table and to produce discrete control outputs. In computer simulation, an ASE controller and a couple of ASE-CMAC neural network are trained to balance the inverted pendulum on a cart. The number of trials until the controllers are established and the learning performance of the controllers are evaluated to find that generalization ability of the CMAC improves the speed of the ASE-reinforcement learning enough to realize the cartpole control system.

Keywords

References

  1. L. P. Kaelbling, M. L. Littman, and A. W. Moore, 'Reinforcement Learning: A Survey,' Journal of Artificial Intelligence Research, Vol. 4, pp. 237-285, 1996
  2. L. Zhao and Z. Liu, 'A Genetic Algorithm for Reinforcement Learning,' Proceedings of the IEEE International Conference on Neural Networks, Vol. 2, pp. 1056-1060,3-6 June 1996
  3. A. G. Barto, R. S. Sutton, and C. W. Anderson, 'Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems,' IEEE Transactions on Systems, Man, and Cybernetics, Vol. SMC-13, No.5, pp. 834-846, September/ October 1983 https://doi.org/10.1109/TSMC.1983.6313077
  4. V.J. Gullapalli, A. Franklin, and H. Benbrahim, 'Acquiring Robot Skills via Reinforcement Learning,' IEEE Control Systems, pp. 13-24, February 1994
  5. T. Kondo and K. Ito, 'A Reinforcement Learning using Adaptive State Space Construction Strategy for Real Autonomous Mobile Robots,' Proceedings of the 41st SICE Annual Conference, Vol. 5, pp. 3139-3144, 5-7 August, 2002
  6. M. Ricordeau, 'Q-Concept-Learning: Generalization with Concept Lattice Representation in Reinforcement Learning,' Proceedings of the 15th IEEE International Conference on Tools with Artificial Intelligence, pp. 316-323, 3- 5 November 2003
  7. D. Michie and R. A. Chambers, ''BOXES' as a Model of Pattern-Formation,' Towards a Theoretical Biology, pp. 206-215, 1968
  8. Y. Hu and R. Fellman, 'A Hardware Efficient Implementation of a Boxes Reinforcement Learning System,' Proceedings of IEEE International Conference on Neural Networks, Vol. 4, pp. 2297-2302, 7 June - 2 July 1994
  9. M. Han and B. Zhang, 'Control of Robotic Manipulators using a CMAC-Based Reinforcement Learning System,' Proceedings of the IEEE/RSJ/GI international Conference on Intelligent Robots and Systems 1994, Volume 3, 2-16, pp. 2117-2122, September 1994
  10. M. T. Rosenstein and A. G. Barto, 'Reinforcement Learning with Supervision by a Stable Controller,' Proceedings of the 2004 American Control Conference, Boston, Massachusetts, pp. 4517-4522, June 30 - July 2, 2004
  11. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, The MIT Press, Cambridge, Massachusetts, 1998
  12. C. Lin and H. Kim, 'CMAC-Based Adaptive Critic Self-Learning Control,' IEEE Transactions on Neural Networks, Vol. 2, No.5, pp. 530-533, September 1991 https://doi.org/10.1109/72.134290
  13. R. S. Sutton, 'Generalization in Reinforcement Learning: Successful Examples using Sparse Coarse Coding,' Advances in Neural information Processing Systems 8, pp. 1038-1-44, 1996
  14. X. Xu, D. Hu, and H. He, 'Accelerated Reinforcement Learning Control using Modified CMAC Neural Networks,' Proceedings of the 9th international Conference on Neural Information Processing, Vol. 5, pp. 2575-2578, 2002
  15. Y. Wei and M. Zhao, 'Effective Strategies for Complex Skill Real-time Learning using Reinforcement Learning,' Proceedings of the 2003 IEEE international Conference on Robotics, Intelligent Systems and Signal Processing, pp. 388-392, October 2003
  16. D. E. Thompson and S. Kwon, 'Neighborhood Sequential and Random Training Techniques for CMAC,' IEEE Transactions on Neural Networks, Vol. 6, No. 1, pp. 196-202, January 1995 https://doi.org/10.1109/72.363437
  17. J. S. Albus, 'Data Storage in the Cerebellar Model Articulation Controller (CMAC),' Journal of Dynamic Systems, Measurement and Control, Transactions ASME, Series G, Vol. 97, No.3, pp. 228-233, September 1975 https://doi.org/10.1115/1.3426923
  18. S. Geva and J. Sitte, 'A Cartpole Experiment Benchmark for Trainable Controllers,' IEEE Control Systems, pp. 40-51, October 1993