Journal of KIISE:Software and Applications (한국정보과학회논문지:소프트웨어및응용)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지

Restructuring a Feed-forward Neural Network Using Hidden Knowledge Analysis

학습된 지식의 분석을 통한 신경망 재구성 방법

  • 김현철 (고려대학교 사범대학 컴퓨터교육과)
  • Published : 2002.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

It is known that restructuring feed-forward neural network affects generalization capability and efficiency of the network. In this paper, we introduce a new approach to restructure a neural network using abstraction of the hidden knowledge that the network has teamed. This method involves extracting local rules from non-input nodes and aggregation of the rules into global rule base. The extracted local rules are used for pruning unnecessary connections of local nodes and the aggregation eliminates any possible redundancies arid inconsistencies among local rule-based structures. Final network is generated by the global rule-based structure. Complexity of the final network is much reduced, compared to a fully-connected neural network and generalization capability is improved. Empirical results are also shown.

다층신경회로망 구조의 재구성은 회로망의 일반화 능력이나 효율성의 관점에서 중요한 문제로 연구되어왔다. 본 논문에서는 신경회로망에 학습된 은닉 지식들을 추출하여 조합함으로써 신경회로망의 구조를 재구성하는 새로운 방법을 제안한다. 먼저, 각 노드별로 학습된 대표적인 지역 규칙을 추출하여 각 노드의 불필요한 연결구조들을 제거한 후, 이들의 논리적인 조합을 통하여 중복 또는 상충되는 노드와 연결구조를 제거한다. 이렇게 학습된 지식을 분석하여 노드와 연결구조를 재구성한 신경회로망은 처음의 신경회로망에 비하여 월등히 감소된 구조 복잡도를 가지며 일반적으로 더 우수한 일반화 능력을 가지게 됨을 실험결과로서 제시하였다.

Keywords

References

  1. Mozer, M. and Smolensky, P., 'Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment,' Advances in Neural Information Processing System 1, (editor: Touretzky, D.S.), 1989
  2. Krogh, A. and Hertz, J.A., 'A Simple Weight Decay Can Improve Generalization,' Advances in Neural Information Processing Systems, Vol.4 (1992) pp.950-957, 1992
  3. Kruschke, J.K., and Movellan, J.R, 'Benefits of Gain: Speeded Learning and Minimal Hidden Layers In Backpropagation Networks,' IEEE Transactions on Systems, Man and Cybernetics, Vol.21(1,) 1991 https://doi.org/10.1109/21.101159
  4. Fu, LiMin and Kim, Hyeoncheol., 'Abstraction and Representation of Hidden Knowledge In an Adapted Neural Network,' unpublished, CISE, University of Florida, 1994
  5. Kim, Hyeoncheol and Fu, LiMin., 'Generalization and Fault Tolerance In Rule-based Neural Network,' In Proceedings of IEEE International Conference on Neural Networks, Vol.5 pp.1550-1555, 1994 https://doi.org/10.1109/ICNN.1994.374386
  6. Opitz, D.W., and Shavlik, J.W., 'Using Genetic Search to Refine Knowledge-Based Neural Networks,' Machine Learning: Proceedings of the 11th International Conference, 1994
  7. Towell, Geoffrey G., Shavlik, Jude W. and Noordewier, M.O., 'Refinement of Approximate Domain Theories by Knowledge-based Neural Networks.' Proceedings AAAI-.90, pp.861-866, 1990
  8. Kim, Hyeoncheol., 'Neural Network Refinement using Hidden Knowledge Extraction,' Journal of KISS: Software and Applications, Vol.27 pp.1082-1087, 2000
  9. Fu, LiMin., Neural Networks in Computer Intelligence, McGraw Hill, Inc., 1994
  10. Andrews, Robert, Diederich, Joachim, Tickle, Alam B., 'Survey and critique of techniques for extracting rules from trained artificial neural networks,' Knowledge-Based Systems, Vol. 8(6) pp.373-389, 1995 https://doi.org/10.1016/0950-7051(96)81920-4
  11. Fu, LiMin., 'Knowledge-based connectionism for revising domain theories,' IEEE Transactions on Systems, Man, and Cybernetics, Vol 23(1) pp.173182, 1993 https://doi.org/10.1109/21.214775
  12. Fu, LiMin., 'Rule generation from neural networks,' IEEE Transactions on Systems, Man, and Cybernetics, Vol. 24(8) pp.1114-1124, 1994 https://doi.org/10.1109/21.299696
  13. Gallant, S.I., 'Connectionist expert systems,' Communications of the ACM, Vol. 31(2) pp.152-169, 1988 https://doi.org/10.1145/42372.42377
  14. Kim, Hyeoncheol.. 'Computationally Efficient Heuristics for If-Then Rule Extraction from Feed-Forward Neural Networks,' Lecture Notes in Artificial Intelligence, Vol. 1967 pp.170-182, 2000
  15. Setino, Rudy, Liu, Huan, 'Understanding neural networks via rule extraction,' Proceedings of the 14th International Conference on Neural Networks, pp.480-485, 1995
  16. Taha, Ismali A. and Ghosh, Joydeep, 'Symbolic interpretation of artificial neural networks,' IEEE Transactions on Knowledge and Data Engineering, Vol. 11(3) pp.443-463, 1999 https://doi.org/10.1109/69.774103
  17. Towell, Geoffrey G. and Shavlik, Jude W., 'Extracting refined rules from knowledge-based neural networks,' Machine Learning, Vol. 13(1), 1993 https://doi.org/10.1007/BF00993103
  18. Towell, Geoffrey G. and Shavlik, Jude W., 'Knowledge-Based Artificial Neural Networks,' Artificial Intelligence, Vol 70, pp.119-165, 1994 https://doi.org/10.1016/0004-3702(94)90105-8
  19. Fu, LiMin., 'Introduction to knowledge-based neural networks,' Knowledge-Based Systems, Vol. 8(6) pp.299-300, 1995 https://doi.org/10.1016/0950-7051(96)81914-9