전자공학회논문지CI (Journal of the Institute of Electronics Engineers of Korea CI)

대한전자공학회 (The Institute of Electronics and Information Engineers)
권호 표지

역전파 알고리즘의 성능개선을 위한 학습율 자동 조정 방식

Auto-Tuning Method of Learning Rate for Performance Improvement of Backpropagation Algorithm

  • 발행 : 2002.07.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

역전파 알고리즘의 성능 개선을 위해서 학습율을 자동 조정하는 방식을 제안하였다. 제안한 방식은 각각의 연결강도의 학습율을 퍼지 논리 시스템을 이용하여 자동 조정하는 방식으로 각각의 연결강도에 대해서 ${\Delta}$와 $\bar{{\Delta}}$를 구하여 퍼지 논리 시스템의 입력으로 사용하고, 학습율을 출력으로 사용하였다. 제안한 방식을 N-패리티 문제, 함수 근사, 숫자 패턴 분류에 대한 시뮬레이션 결과 일반적인 역전파 알고리즘, 모멘텀 방식, Jacobs의 delta-bar-delta 방식보다 성능이 개선됨을 확인하였다.

We proposed an auto-tuning method of learning rate for performance improvement of backpropagation algorithm. Proposed method is used a fuzzy logic system for automatic tuning of learning rate. Instead of choosing a fixed learning rate, the fuzzy logic system is used to dynamically adjust learning rate. The inputs of fuzzy logic system are ${\Delta}$ and $\bar{{\Delta}}$, and the output is the learning rate. In order to verify the effectiveness of the proposed method, we performed simulations on a N-parity problem, function approximation, and Arabic numerals classification. The results show that the proposed method has considerably improved the performance compared to the backpropagation, the backpropagation with momentum, and the Jacobs' delta-bar-delta.

키워드

참고문헌

  1. Martin T. Hagan, Howard B. Demuth, Mark Beale, Neural Network Design, PWS Publishing, Boston, 1995
  2. Peiman G. Maghami and Dean W.Sparks, 'Design of Neural Networks for Fast Convergence and Accuracy:Dynamics and Control', IEEE Transactions on Neural Networks, Vol. 11, No. 11, pp. 113-123, January 2000 https://doi.org/10.1109/72.822515
  3. Tokumitsu Fujita,Takao Watanebe and Keiichiro Yasuda, 'A Study on Improvement in Learning Efficiency of Multilayered Neural Networks based on Dynamical System', T. IEE Japan, Vol. 117-C, No. 12, pp. 1848-1855, 1997
  4. B.Hassibi, D.G.Stork and G.H.Wolff, 'Optimal brain surgeon and general network puring', Proceedings of the IEEE International Joint Conference on Neural Networks, Vol. 2, pp. 441-444, 1992
  5. A.K.Rigler, J.M. Irvine and T.P. Vogl, 'Rescaling of variables in backpropagation learning', Neural Networks, Vol. 3, No. 5, pp. 561∼573, 1990 https://doi.org/10.1016/0893-6080(90)90006-7
  6. R. A. Jacobs, 'Increased rates of convergence through learning rate adaptation,' Neural Networks, Vol. 1, No. 4, pp. 295-308, 1988 https://doi.org/10.1016/0893-6080(88)90003-2
  7. M. T. Hagan and M. Menhaj, 'Training feedforward networks with the Marquardt algorithm,' IEEE Tran. on Neural Networks, Vol. 5, No.6, 1994 https://doi.org/10.1109/72.329697
  8. C.Charalambous, 'Conjugate gradient algorithm for efficient training of artificial neural networks,' IEE Proceedings, Vol. 139, No. 3, pp. 301-310, 1992
  9. Li-Xin Wang, A Course in fuzzy systems and control, Prentice-Hall,Inc., 1997