The KIPS Transactions:PartB (정보처리학회논문지B)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Reinforcement Loaming Method using TD-Error in Ant Colony System

개미 집단 시스템에서 TD-오류를 이용한 강화학습 기법

  • 이승관 (경희대학교 대학원 전자계산공학과) ;
  • 정태충 (경희대학교 컴퓨터공학과)
  • Published : 2004.02.01
Download PDF
⟨ Previous Next ⟩

Abstract

Reinforcement learning takes reward about selecting action when agent chooses some action and did state transition in Present state. this can be the important subject in reinforcement learning as temporal-credit assignment problems. In this paper, by new meta heuristic method to solve hard combinational optimization problem, examine Ant-Q learning method that is proposed to solve Traveling Salesman Problem (TSP) to approach that is based for population that use positive feedback as well as greedy search. And, suggest Ant-TD reinforcement learning method that apply state transition through diversification strategy to this method and TD-error. We can show through experiments that the reinforcement learning method proposed in this Paper can find out an optimal solution faster than other reinforcement learning method like ACS and Ant-Q learning.

강화학습에서 temporal-credit 할당 문제 즉, 에이전트가 현재 상태에서 어떤 행동을 선택하여 상태전이를 하였을 때 에이전트가 선택한 행동에 대해 어떻게 보상(reward)할 것인가는 강화학습에서 중요한 과제라 할 수 있다. 본 논문에서는 조합최적화(hard combinational optimization) 문제를 해결하기 위한 새로운 메타 휴리스틱(meta heuristic) 방법으로, greedy search뿐만 아니라 긍정적 반응의 탐색을 사용한 모집단에 근거한 접근법으로 Traveling Salesman Problem(TSP)를 풀기 위해 제안된 Ant Colony System(ACS) Algorithms에 Q-학습을 적용한 기존의 Ant-Q 학습방범을 살펴보고 이 학습 기법에 다양화 전략을 통한 상태전이와 TD-오류를 적용한 학습방법인 Ant-TD 강화학습 방법을 제안한다. 제안한 강화학습은 기존의 ACS, Ant-Q학습보다 최적해에 더 빠르게 수렴할 수 있음을 실험을 통해 알 수 있었다.

Keywords

References

  1. A. Colorni, M. Dorigo and V. Maniezzo, 'An investigation of some properties of an ant algorithm,' Proceediings of the Parallel Parallel Problem Solving from Nature Conference(PPSn'92), R. Manner and B. Manderick (Eds.), Elsevier Publishing, pp.509-520, 1992
  2. A. Colorni, M. Dorigo and V. Maniezzo, 'Distributed optimization by ant colonies', Proceedings of ECAL'91 - Eu - ropean Conference fo Artificial Life, Paris, France, F. Varela and P. Bourgine (Eds.), Elsevier Publishing, pp.134-144, 1991
  3. C. J. C. H. Watkins, 'Learning from Delayed Rewards, King's College,' Ph.D. thesis, King's College, Cambrige, U.K, 1989
  4. C. N. Fiecher, 'Efficient reinforcement learning', In Proceedings of the Seventh Annual ACM Conference On Computational Learning THeory, pp.88-97, 1994
  5. E. Barnald, 'Temporal-difference methods and markov model,' IEEE Transactions on Systems, Man, and Cybernetics, Vol.23, pp.357-365, 1993 https://doi.org/10.1109/21.229449
  6. L. M. Gambardella and M. Dorigo, 'Solving symmetric and asymmetric TSPs by ant colonies', Proceedings of IEEE International Conference of Evolutionary Computation, IEEE-EC'96, IEEE Press, pp.622-627, 1996 https://doi.org/10.1109/ICEC.1996.542672
  7. L. M. Gambardella and M. Dorigo, 'Ant Colony System : A Cooperative Learning apprach to the Traveling Salesman Problem', IEEE Transactions on Evolutionary Computation, Vol.1, No.1, 1997 https://doi.org/10.1109/4235.585892
  8. L. M. Gambradella and M. Dorigo, 'Ant-Q : a reinforcement learning approach to the traveling salesman problem', Proceedings of ML-95, Twelfth International Conference on Machine Learning, A. Prieditis and S. Russell (Eds.), Morgan Kaufmann, pp.252-260, 1995
  9. M. Dorigo and L. M. Gambardella, 'A study of some properties of Ant-Q', Proceedings of PPSN IVFourth International Conference on Parallel Problem Solving From Nature, H. M.Voigt, W. Ebeling, I. Rechenberg and H. S. Schwefel (Eds.), Springer-Verlan, Berlin, pp.656-665, 1996
  10. M. Drigo and V. Maniezzo and A. Colorni, 'The ant system : optimization by a colony of cooperation agents', IEEE Transactions of Systems, Man and Cybernetics- Part B, Vol.26, No.2, pp.29-41, 1996 https://doi.org/10.1109/3477.484436
  11. M. Dorigo and G. D. Caro, 'Ant Algorithms for Discrete Optimization', Artificial Life, Vol.5, No.3, pp.137-172, 1999 https://doi.org/10.1162/106454699568728
  12. M. Dorigo and L. M. Gambardella, 'Ant Colonies for the Traveling Salesman Problem', BioSystems, 43, pp.73-81, 1997 https://doi.org/10.1016/S0303-2647(97)01708-5
  13. R. C. Yee, P. E. Utgoff and A. G. Barto, 'Explaining temporal differences to create useful concepts for evaluating states', In Proceedings of the 8th National Conference on Artificial Intelligence, pp.882-888, 1990
  14. T. Stutzle, and H. Hoos, 'The ant system and local search for the traveling salesman problem', Proceedings of ICEC 1997-1997 IEEE 4th International Conference fo Evolutionary
  15. T. Sttzle and M. Dorigo, 'ACO Algorithms for the Traveling Saleslman Problem,' In K. Miettinen, M. Makela, P. Neittaanmaki, J. Periaux, editors, Evolutionary Algorithms in Engineering and Computer Science, Wiley, 1999