The Performance Improvement of Backpropagation Algorithm using the Gain Variable of Activation Function

활성화 함수의 이득 가변화를 이용한 역전파 알고리즘의 성능개선

  • Published : 2001.11.25

Abstract

In order to improve the several problems of the general backpropagation, we propose a method using a fuzzy logic system for automatic tuning of the activation function gain in the backpropagation. First, we researched that the changing of the gain of sigmoid function is equivalent to changing the learning rate, the weights, and the biases. The inputs of the fuzzy logic system were the sensitivity of error respect to the last layer and the mean sensitivity of error respect to the hidden layer, and the output was the gain of the sigmoid function. In order to verify the effectiveness of the proposed method, we performed simulations on the parity problem, function approximation, and pattern recognition. The results show that the proposed method has considerably improved the performance compared to the general backpropagation.

일반적인 역전파 알고리즘의 여러 가지 문제점들을 개선하기 위하여 활성화 함수의 이득을 퍼지 로직 시스템을 이용하여 자동 조절하는 방식을 제안하였다. 퍼지 로직 시스템을 구성하기 위하여 먼저 활성화 함수의 이득의 변화가 학습율, 연결강도 바이어스 등의 변화와 등가인 관계를 조사하였다 퍼지 로직 시스템의 입력은 마지막층에 대한 오차의 감도와 은닉층에 대한 오차의 평균 감도를 사용하였고, 출력은 활성화 함수의 이득을 사용하였다. 제안한 방식과 일반적인 역전파 알고리즘을 패리티 문제, 함수 근사화 문제 및 패턴 인식 문제등에 대하여 시뮬레이션하여 비교 검토한 결과 수렴비, 평균 학습 반복수, 정말도 및 새로운 입력 에 대한 원하는 오차 범위의 출력을 얻는 등의 성능이 개선됨을 알았다.

Keywords

References

  1. Peiman G. Maghami and Dean W.Sparks, 'Design of Neural Networks for Fast Convergence and Accuracy:Dynamics and Control', IEEE Transactions on Neural Networks, Vol.11, No.11, pp.113-23, January 2000 https://doi.org/10.1109/72.822515
  2. Martin T. Hagan, Howard B. Demuth, Mark Beale, Neural Network Design, PWS Publishing, Boston, 1995
  3. Tokumitsu Fujita,Takao Watanebe and Keiichiro Yasuda, 'A Study on Improvement in Learning Efficiency of Multilayered Neural Networks based on Dynamical System', T.IEE Japan, Vol.117-C, No.12, pp.1848-1855, 1997
  4. A. Sperduti and A.Starita, 'Speed up Learning and Network Optimization with Extended Backpropagation', Neural Networks, Vol.6, pp. 365-383, 1993 https://doi.org/10.1016/0893-6080(93)90004-G
  5. B.Hassibi, D.G.Stork and G.H.Wolff, 'Optimal brain surgeon and general network puring', Proceedings of the IEEE International Joint Conference on Neural Networks, vol.2, pp.441-444, 1992 https://doi.org/10.1109/ICNN.1993.298572
  6. A.K.Rigler, J.M. Irvine and T.P. Vogl,'Rescaling of variables in backpropagation learning', Neural Networks, vol.3, no. 5, pp. 561-573, 1990 https://doi.org/10.1016/0893-6080(91)90006-Q
  7. T.H.Cho, R.W.Conners and P.A. Araman, 'Fast Back-propagation Learning using Steep Activation Functions and Automatic Weight Reinitialization', Proc. IEEE Int. Conf. Systems, Man, Cybernetics:Decision Aiding for Complex Syst. 3, pp. 1587-1592, 1991 https://doi.org/10.1109/ICSMC.1991.169915
  8. R. A. Jacobs, 'Increased rates of convergence through learning rate adaptation,' Neural Networks, vol.1, no.4, pp. 295-308, 1988 https://doi.org/10.1016/0893-6080(88)90003-2
  9. M. T. Hagan and M. Menhaj, 'Training feedforward networks with the Marquardt algorithm,' IEEE Tran. on Neural Networks, vol.5, no.6, 1994 https://doi.org/10.1109/72.329697
  10. C. Charalambous, 'Conjugate gradient algorithm for efficient training of artificial neural networks,' IEE Proceedings, vol.139, no.3, pp.301-310, 1992
  11. Sanossian, H. Y. Y. and Evans, D. J., 'Gradient Range-Based Heuristic Method for Accelerating Neural Network Convergence,' Integrated Computer-Aided Engineering, Vol.2(2), pp.147-152, 1995
  12. C. Codrington and M. Tenorio, 'Adaptive gain networks', Proc. IEEE, Int. Conf. Neural Networks, ICNN94, 1, pp. 339-344, 1994 https://doi.org/10.1109/ICNN.1994.374186
  13. J. M. Zurada, Introduction to Artificial Neural Systems, West Publishing Company, St. Paul, MN, 1992
  14. W. Pedrycz, Fuzzy control and Fuzzy systems, John Wiley& Sons, Inc., 1992