A self-organizing algorithm for multi-layer neural networks

다층 신경회로망을 위한 자기 구성 알고리즘

  • 이종석 (한국과학기술원 전자전산학과) ;
  • 김재영 (한국과학기술원 전자전산학) ;
  • 정승범 (삼성전자(주)기술총괄소프트웨어센) ;
  • 박철훈 (한국과학기술원 전자전산학과)
  • Published : 2004.05.01

Abstract

When a neural network is used to solve a given problem it is necessary to match the complexity of the network to that of the problem because the complexity of the network significantly affects its learning capability and generalization performance. Thus, it is desirable to have an algorithm that can find appropriate network structures in a self-organizing way. This paper proposes algorithms which automatically organize feed forward multi-layer neural networks with sigmoid hidden neurons for given problems. Using both constructive procedures and pruning procedures, the proposed algorithms try to find the near optimal network, which is compact and shows good generalization performance. The performances of the proposed algorithms are tested on four function regression problems. The results demonstrate that our algorithms successfully generate near-optimal networks in comparison with the previous method and the neural networks of fixed topology.

신경회로망을 이용하여 주어진 문제를 해결할 때, 문제의 복잡도에 맞는 구조를 찾는 것이 중요하다. 이것은 신경회로망의 복잡도가 학습능력과 일반화 성능에 크게 영향을 주기 때문이다. 그러므로, 문제에 적합한 신경회로망의 구조를 자기 구성적으로 찾는 알고리즘이 유용하다. 본 논문에서는 시그모이드 활성함수를 가지는 전방향 다층 신경회로망에 대하여 주어진 문제에 맞는 구조를 결정하는 알고리즘을 제안한다. 개발된 알고리즘은 구조증가 알고리즘과 연결소거 알고리즘을 이용하여, 주어진 학습 데이터에 대해 가능한 한 작은 구조를 가지며 일반화 성능이 좋은 최적에 가까운 신경회로망을 찾는다. 네 가지 함수 근사화 문제에 적용하여 알고리즘의 성능을 알아본다. 실험 결과에서, 제안한 알고리즘이 기존의 알고리즘 및 고정구조를 갖는 신경회로망과 비교하였을 때 최적 구조에 가까운 신경회로망을 구성하는 것을 확인한다.

Keywords

References

  1. S. Haykin Neural Networks: A Comprehensive Foundation, NJ: Prentice-Hall, 1999
  2. M. T. Hagan, H. B. Dermuth, and M. Beale, Neural Network Design, Boston, MA: PWS Publishing, 1996
  3. T.-Y. Kwok and D.-Y. Yeung, 'Constructive algorithms for structure learning in feedforward neural networks for regression problems,' IEEE Trans. Neural Networks, vol. 8, no. 3, pp. 630-645 https://doi.org/10.1109/72.572102
  4. T. Ash, 'Dynamic node creation in backpropagation networks,' Connection Sci., vol. 1, no. 4, pp. 365-375, 1989 https://doi.org/10.1080/09540098908915647
  5. R. Setiono and L. C. K. Hui, 'Use of a quasi-Newton method in a feedforward neural network construction algorithm,' IEEE Trans. Neural Networks, vol. 6, no. 1, pp. 273-277, Jan. 1995 https://doi.org/10.1109/72.363426
  6. M. T. Hagan and M. B. Menhaj, 'Training feedforward networks with the Marquardt algorithm,' IEEE Trans. Neural Networks, vol. 5, no. 6, pp. 989-993, Nov. 1994 https://doi.org/10.1109/72.329697
  7. J.-S. Lee and C. H. Park, 'Self-organizing neural network using adaptive neurons,' in Proc. Int. Conf. Neural Information Processing, Singapore, pp. 935-939, Nov. 2002
  8. K. Hormik, M. Stinchdombe, and H. White, 'Multilayer feedforward networks are universal approximaters,' Neural Networks, vol. 2, pp. 359-366, 1989 https://doi.org/10.1016/0893-6080(89)90020-8
  9. S. Tamura and M. Tateishi, 'Capabilities of a four-layered feedforward neural network: four layers versus three,' IEEE Trans. Neural Networks, vol. 8, pp. 251-255, Mar. 1997 https://doi.org/10.1109/72.557662
  10. L. Breiman 'The pi method for estimating multivariate functions from noisy data,' Technometrics, vol. 3, no. 2, pp. 125-160, 1991 https://doi.org/10.2307/1269038
  11. M. Maechler, D. Martin, J. Schimert, M. Csoppensky and J. N. Hwang, 'Project pursuit learning networks for regression,' in Proc. Int. Conf. Tools for AI, Washington D.C., pp. 350-358, Nov. 1990 https://doi.org/10.1109/TAI.1990.130362
  12. V. Cherkassly and H. Lari-Najafi, 'Constrained topological mapping for nonparametric regression analysis,' Neural Networks, vol. 4, pp. 27-40, 1991 https://doi.org/10.1016/0893-6080(91)90028-4
  13. J. H. Friedman, 'Classification and multiple regression through projection pursuit,' Dept. Statistics, Standford Univ. Technical Report, no12, Jan. 1985