Journal of the Institute of Electronics Engineers of Korea CI (전자공학회논문지CI)

The Institute of Electronics and Information Engineers (대한전자공학회)
권호 표지

Genetic Algorithm for Node P겨ning of Neural Networks

신경망의 노드 가지치기를 위한 유전 알고리즘

  • Heo, Gi-Su (Department of Computer and Information Science, Chonbuk National University) ;
  • Oh, Il-Seok (Department of Computer Engineering, Chonbuk National University)
  • 허기수 (전북대학교 컴퓨터정보학과) ;
  • 오일석 (전북대학교 전자정보공학부 컴퓨터공학)
  • Published : 2009.03.25
Download PDF
⟨ Previous Next ⟩

Abstract

In optimizing the neural network structure, there are two methods of the pruning scheme and the constructive scheme. In this paper we use the pruning scheme to optimize neural network structure, and the genetic algorithm to find out its optimum node pruning. In the conventional researches, the input and hidden layers were optimized separately. On the contrary we attempted to optimize the two layers simultaneously by encoding two layers in a chromosome. The offspring networks inherit the weights from the parent. For teaming, we used the existing error back-propagation algorithm. In our experiment with various databases from UCI Machine Learning Repository, we could get the optimal performance when the network size was reduced by about $8{\sim}25%$. As a result of t-test the proposed method was shown better performance, compared with other pruning and construction methods through the cross-validation.

신경망의 구조를 최적화하기 위해서는 노드 또는 연결을 잘라내는 가지치기 방법과 노드를 추가해 나가는 구조 증가 방법이 있다. 이 논문은 신경망의 구조 최적화를 위해 가지치기 방법을 사용하며, 최적의 노드 가지치기를 찾기 위해 유전 알고리즘을 사용한다. 기존 연구에서는 입력층과 은닉층의 노드를 따로 최적화 대상으로 삼았다 우리는 두 층의 노드를 하나의 염색체에 표현하여 동시 최적화를 꾀하였다. 자식은 부모의 가중치를 상속받는다 학습을 위해서는 기존의 오류 역전파 알고리즘을 사용한다. 실험은 UCI Machine Learning Repository에서 제공한 다양한 데이터를 사용하였다. 실험 결과 신경망 노드 가지치기 비율이 평균 $8{\sim}25%$에서 좋은 성능을 얻을 수 있었다. 또한 다른 가지치기 및 구조 증가 알고리즘과의 교차검증에 대한 t-검정 결과 그들에 비해 우수한 성능을 보였다.

Keywords

References

  1. L. Fausett, Fundamentals of Neural Networks, Prentice Hall, 1994
  2. X. Yao, 'Evolving Artificial Neural Networks,' Proc. of the IEEE, Vol. 87, No. 9, pp. 1423-1447, 1999 https://doi.org/10.1109/5.784219
  3. I.-S. Oh, J.-S. Lee, and B.-R. Moon, 'Hybrid Genetic Algorithms for Feature Selection,' IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 26, No. 11, pp. 1424-1437, 2004 https://doi.org/10.1109/TPAMI.2004.105
  4. G. Castellano and A. M. Fanelli, 'An Iterative Pruning Algorithm for Feedforward Neural Networks,' IEEE Trans. on Neural Networks, Vol. 8, No. 3, pp. 519-531, 1997 https://doi.org/10.1109/72.572092
  5. W. Wang, W. Lu, A. Leung, S. M. Lo, Z. Xu, and X. Wang, 'Optimal Feed-forward Neural Networks Based on the Combination of Constructing and Pruning by Genetic Algorithms,' Proc. of the 2002 Intl. Joint Conf. on Neural Networks, Vol. 1, pp. 636-641, 2002
  6. Z. Guo and R. E. Uhring, 'Using Genetic Algorithms to Select Inputs for Neural Networks,' Intl. Workshop on Combinations of Genetic Algorithms and Neural Networks, pp. 223-234, 1992
  7. 이하준, '임팩트 팩터 정규화 기법을 이용한 신경회로망 연결소거 알고리즘', 한국과학기술원 석사학위 논문, 2001
  8. S. Theodoridis and K. Koutroumbas, Pattern Recognition, Academic Press, 2006
  9. R. Reed, 'Pruning Algorithms - A Survey,' IEEE Trans. on Neural Networks, Vol. 4, No. 5, pp. 740-747, 1993 https://doi.org/10.1109/72.248452
  10. Y. Cun, J. S. Denker, and S. A. Solla, 'Optimal Brain Damage,' Advances in Neural Information Processing Systems 2, pp. 598 - 605, 1990
  11. B. Hassibi and D. G. Stork, 'Second Order Derivatives for Network Pruning: Optimal Brain Surgeon,' Advances in Neural Information Processing Systems 5, pp. 164 - 171, 1992
  12. B. E. Segee and M. J. Carter, 'Fault Tolerance of Pruning Multilayer Networks,' in Proc. Intl. Joint Conf. Neural Networks, Vol. Ⅱ(Seattle), pp. 447-452. 1991
  13. E. D. Karnin, 'A Simple Procedure for Pruning Back-propagation trained Neural Networks,' IEEE Trans. Neural Networks, Vol. 1, No. 2, pp. 239-242, 1990 https://doi.org/10.1109/72.80236
  14. J. Branke, 'Evolutionary Algorithms for Neural Network Design and Training,' Proc. of the First Nordic Workshop on Genetic Algorithms and their Applications, 1995
  15. G. Miller, P. Todd, and S. Hedge, 'Designing Neural Networks using Genetic Algorithms,' Proc. 3rd Intl. Joint Conf. on Genetic Algorithms, pp. 379-384, 1989
  16. P. Arena, R. Caponetto, L. Fortuna, and M. G. Xibilia, 'Genetic Algorithm to Select Optimal Neural Network Topology,' Proc. 35th Midwest Symposium on Circuit and Systems, Washington, 1992
  17. P. Arena, R. Caponetto, L. Fortuna, and M. G. Xibilia, 'MLP Optimal Selection Via Genetic Algorithms,' Proc. Intl. Conf. Neural Networks and Genetic Algorithms, Innsbruck, Austria, 1993
  18. J. C .F. Pujol and R. Poli, 'Evolving the Topology and the Weights of Neural Networks Using a Dual Representation,' Applied Intelligence, Vol. 8, No. 1, pp. 73-84, 1998 https://doi.org/10.1023/A:1008272615525
  19. Y. LeCun, 'Generalization and Network Design Strategies,' Technical Report CRG-TR-89-4, Department of Computer Science, University of Toronto, Canada, 1989
  20. P. Koehn, 'Combining Genetic Algorithms and Neural Networks: The Encoding Problem,' Master's thesis, University of Tennessee, Knoxville, 1994
  21. H. Kitano, 'Designing Neural Networks Using Genetic Algorithms with Graph Generation System,' Complex Systems, Vol. 4, pp. 461-467, 1990
  22. F. Gruau, 'Neural Networks Synthesis Using Cellular Encoding and The Genetic Algorithm,' PhD thesis, Laboratoire de l’ Informatique du Parallilisme, Ecole Normale Supirieure de Lyon, France, 1994
  23. 문병로, 유전 알고리즘, 두양사, 2003
  24. http://mlearn.ics.uci.edu/MLRepository.html
  25. Neural Network FAQ, 2004 [online] Available: ftp://ftp.sas.com/pub/neural/FAQ.html , W. S. Sarles, periodic posting to the Usenet newsgroup comp.ai.neural-net, URL
  26. P. P. Palmes, T. Hayasaka, and S. Usui, 'Mutation-Based Genetic Neural Network,' IEEE Trans. on Neural Networks, Vol. 16, No. 3, pp. 587-600, 2005 https://doi.org/10.1109/TNN.2005.844858
  27. T. Q. Huynh and R. Setiono, 'Effective Neural Network Pruning Using Cross-validation,' Proc. Intl. Joint Conf. on Neural Networks, pp. 972-977, 2005
  28. R. Quinlan, 'C4.5: Programs for Machine Learning,' Morgan Kanufman, San Mateo, CA, 1993
  29. E. Frank, Y. Wang, S. Inglis, G. Holmes and I. H. Witten, 'Using Model Trees for Classification,' Machine Learning, Vol. 32, No. 1, pp. 63-76, 1997 https://doi.org/10.1023/A:1007421302149
  30. R. Setiono, 'Feedforward Neural Network Construction Using Cross-validation,' Neural Computation, Vol. 13, No. 12. pp. 2865-2877, 2001 https://doi.org/10.1162/089976601317098565