Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Layer-by-Layer Learning Algorithm using Correlation Coefficient for Multilayer Perceptrons

상관 계수를 이용한 다층퍼셉트론의 계층별 학습

  • Kwak, Young-Tae (Division of Information Technology, Chonbuk National University)
  • 곽영태 (전북대학교 IT정보공학부)
  • Received : 2011.04.12
  • Accepted : 2011.05.11
  • Published : 2011.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Ergezinger's method, one of the layer-by-layer algorithms used for multilyer perceptrons, consists of an output node and can make premature saturations in the output's weight because of using linear least squared method in the output layer. These saturations are obstacles to learning time and covergence. Therefore, this paper expands Ergezinger's method to be able to use an output vector instead of an output node and introduces a learning rate to improve learning time and convergence. The learning rate is a variable rate that reflects the correlation coefficient between new weight and previous weight while updating hidden's weight. To compare the proposed method with Ergezinger's method, we tested iris recognition and nonlinear approximation. It was found that the proposed method showed better results than Ergezinger's method in learning convergence. In the CPU time considering correlation coefficient computation, the proposed method saved about 35% time than the previous method.

다층퍼셉트론의 계층별 학습 방법의 하나인 Ergezinger 방법은 출력 노드가 1개로 구성되어 있고, 출력층의 가중치를 최소자승법으로 학습하기 때문에 출력층의 가중치에 조기포화 현상이 발생할 수 있다. 이런 조기 포화현상은 학습 시간과 수렴 속도에 장애가 된다. 따라서, 본 논문은 Ergezinger의 학습 방법을 출력층에서 벡터 형태로 학습할 수 있는 알고리즘으로 확대하고 학습 시간과수렴 속도를 개선하기 위해서 학습 상수를 도입한다. 학습상수는 은닉층 가중치 조정 시, 새로이 계산된 가중치와 기존 가중치의 상관 관계를 계산하여 학습 상수에 반영하는 가변적인 방법이다. 실험은 제안된 방법과 기존 방법의 비교를 위해서 iris 문제와 비선형 근사화 문제를 대상으로 실험하였다. 실험에서, 제안 방법은 기존 Ergezinger 방법보다 학습 시간과 수렴 속도에서 우수한 결과를 얻었으며, 상관 관계를 고려한 CPU time 측정에서도 제안한 방법이 기존 방법보다 약 35%의 시간을 절약할 수 있었다.

Keywords

References

  1. D. E. Rumelhart, and J. L. McClelland, Parallel Distributed Processing, MIT Press, Cambridge, MA, pp. 318-362, 1986.
  2. Kyong Ho Lee, "A Study on the Implementation of Serious Game Learning Multiplication Table using Back Propagation Neural Network on Divided Interconnection Weights Table," Journal of The Korea Society of Computer and Information, Vol. 14, No. 10, pp. 233-240, Oct. 2009.
  3. T. P. Vogal, J. K. Mangis, A. K. Zigler, W. T. Zink ,and D. L. Alkon, "Accelerating the convergence of the backpropa gation method," Biological Cybernetics, Vol. 59, pp. 256-264, Sept. 1988.
  4. T. Tollenaere, "SuperSAB: Fast adaptive back propa gation with good scaling properties," Neural Networks, Vol. 3, No. 5, pp. 561-573, 1990. https://doi.org/10.1016/0893-6080(90)90006-7
  5. M. Kordos andW. Duch, "Variable step search algorit hmfor feedforward networks," Neurocomputing, Vol. 71, pp. 2470-2480, April 2008. https://doi.org/10.1016/j.neucom.2008.02.019
  6. S. Ergezinger, and E. Thomsen, "An accelerated lear ning algorithm for multilayer perceptrons optimizati on Layer by Layer," IEEE Trans. on Neural Networks, Vol. 6, No. 1, pp. 31-42, June 1995. https://doi.org/10.1109/72.363452
  7. Rolnald E. Miller, "Optimization," JohnWiley & Son, INC. pp. 358-362, 2000.
  8. M. T. Hagan, and M. Menhaj, "Training feedforward net works with the Marquardt algorithm," IEEE Trans. on Neural Networks, Vol. 5, No. 6, pp. 989-993, Nov. 1994. https://doi.org/10.1109/72.329697
  9. Young-Tae Kwak, "Accelerating Levenberg- Marqua rdt Algorithm using Variable Damping Parameter," Journal of The Korea Society of Computer and Information, Vol. 15, No. 4, pp. 57-63, April 2010. https://doi.org/10.9708/jksci.2010.15.4.057
  10. C. Charalambous, "Conjugate gradient algorithmfor efficient training of artificial neural networks," IEEE Proceedings, Vol. 139, No. 3, pp. 301-310, 1992.
  11. Wei Chu, Chong Jin Ong, and Keerthi S.S., "An imp roved conjugate gradient scheme to the solution of least squares SVM," IEEE Trans. on Neural Networks, Vol. 16, No. 2, pp. 498-501, March 2005. https://doi.org/10.1109/TNN.2004.841785
  12. B. Ph. van Milligen, V. Tribaldos, J. A. Jimenez, an d C. Santa Cruz, ''Comments on "An accelerated learning algorithm for multilayer perceptrons optimization layer by layer"," IEEE Trans. on Neural Networks, Vol. 9, No. 2, pp. 339-341, March 1998. https://doi.org/10.1109/72.661128
  13. Jim Y. F. Yam, and Tommy W. S. Chow, ''Extended least squares based algorithms for training feedforward networks," IEEE Trans. on Neural Networks, Vol. 8, No. 3, pp. 806-810, May 1997. https://doi.org/10.1109/72.572119
  14. UCI Machine Learning Repository http://archive.ics.uci.edu/ml/
  15. F. Biegler-Kong, and F. Barmann, "Alearning algo rithmfor multilayered neural networks based on linear squares problems," Neural Networks, Vol. 6, pp. 127-131, 1993. https://doi.org/10.1016/S0893-6080(05)80077-2