International Journal of Contents

The Korea Contents Association (한국콘텐츠학회)
권호 표지
DOI QR코드

DOI QR Code

Contour Plots of Objective Functions for Feed-Forward Neural Networks

  • Oh, Sang-Hoon (Department of Information Communication Engineering, Mokwon University)
  • Received : 2012.09.07
  • Accepted : 2012.10.26
  • Published : 2012.12.28
Download PDF
⟨ Previous Next ⟩

Abstract

Error surfaces provide us with very important information for training of feed-forward neural networks (FNNs). In this paper, we draw the contour plots of various error or objective functions for training of FNNs. Firstly, when applying FNNs to classifications, the weakness of mean-squared error is explained with the viewpoint of error contour plot. And the classification figure of merit, mean log-square error, cross-entropy error, and n-th order extension of cross-entropy error objective functions are considered for the contour plots. Also, the recently proposed target node method is explained with the viewpoint of contour plot. Based on the contour plots, we can explain characteristics of various error or objective functions when training of FNNs proceeds.

Keywords

References

  1. K. Hornik, M. Stinchcombe, and H. White, "Multilayer Feed-forward Networks are Universal Approximators," Neural Networks, vol.2, 1989, pp. 359-366. https://doi.org/10.1016/0893-6080(89)90020-8
  2. K. Hornik, "Approximation Capabilities of Multilayer Feedforward Networks," Neural Networks, vol.4, 1991, pp. 251-257 https://doi.org/10.1016/0893-6080(91)90009-T
  3. S. Suzuki, "Constructive Function Approximation by Three-Layer Artificial Neural Networks," Neural Networks, vol.11, 1998, pp. 1049-1058 https://doi.org/10.1016/S0893-6080(98)00068-9
  4. Y. Liao, S. C. Fang, H. L. W. Nuttle, "Relaxed Conditions for Radial-Basis Function Networks to be Universal Approximators," Neural Networks, vol.16, 2003, pp. 1019-1028 https://doi.org/10.1016/S0893-6080(02)00227-7
  5. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing, Cambridge, MA, 1986.
  6. C. Wang and J. C. Princope, "Training Neural Networks with Additive Noise in the Desired Signal," IEEE Trans. Neural Networks, vol.10, 1999, pp. 1511-1517. https://doi.org/10.1109/72.809097
  7. J. B. Hampshare and A. H. Waibel, "A Novel Objective Function for Improved Phoneme Recognition Using Time-Delay Neural Networks," IEEE Trans. Neural Networks, vol.1, 1990, pp. 216-228. https://doi.org/10.1109/72.80233
  8. K. Liano, "Robust Error measure for Supervised Neural Network Learning with Outliers," IEEE Trans. Neural Networks, vol.7, 1996, pp. 246-250. https://doi.org/10.1109/72.478411
  9. A. van Ooyen and B. Nienhuis, "Improving the Convergence of the Backpropagation Algorithm," Neural Networks, vol.5, 1992, pp. 465-471. https://doi.org/10.1016/0893-6080(92)90008-7
  10. S.-H. Oh, "Improving the Error Back-Propagation Algorithm with a Modified Error Function," IEEE Trans. Neural Networks, vol.8, 1997, pp. 799-803. https://doi.org/10.1109/72.572117
  11. S.-H. Oh, "Error Back-Propagation Algorithm for Classification of Imbalanced Data," Neurocomputing, vol.74, 2011, pp. 1058-1061. https://doi.org/10.1016/j.neucom.2010.11.024
  12. H. White, "Learning in Artificial Neural Networks: A Statistical Perspective," Neural Computation, vol.1, no.4, Winter 1989, pp. 425-464. https://doi.org/10.1162/neco.1989.1.4.425
  13. M. D. Richard and R. P. Lippmann, "Neural Network Classifier Estimate Bayesian a Posteriori Probabilities," Neural Computa., vol.3, 1991, pp. 461-483. https://doi.org/10.1162/neco.1991.3.4.461
  14. S.-H. Oh, "Statistical Analyses of Various Error Functions for Pattern Classifiers," Proc. Convergence on Hybrid Information Technology, CCIS vol. 206, 2011, p. 129-133.
  15. S.-H. Oh, "A Statistical Perspective of Neural Networks for Imbalanced Data problems," Int. Journal of Contents, vol.7, 2011, pp. 1-5. https://doi.org/10.5392/IJoC.2011.7.3.001
  16. N. Baba, "A New Approach for Finding the Global Minimum of Error Function of Neural Networks," Neural Networks, vol.2, 1989, pp. 367-373. https://doi.org/10.1016/0893-6080(89)90021-X
  17. Y. Lee, S.-H. Oh, and M. W. Kim, "An Analysis of Premature Saturation in Back-Propagation Learning," Neural Networks, vol.6, 1993, pp. 719-728. https://doi.org/10.1016/S0893-6080(05)80116-9