Method that determining the Hyperparameter of CNN using HS algorithm

HS 알고리즘을 이용한 CNN의 Hyperparameter 결정 기법

  • Lee, Woo-Young (Department of Electrical and Electronics Engineering, Chung-Ang University) ;
  • Ko, Kwang-Eun (Department of Electrical and Electronics Engineering, Chung-Ang University) ;
  • Geem, Zong-Woo (Department of Energy IT, Gachon University) ;
  • Sim, Kwee-Bo (Department of Electrical and Electronics Engineering, Chung-Ang University)
  • 이우영 (중앙대학교 전자전기공학부) ;
  • 고광은 (중앙대학교 전자전기공학부) ;
  • 김종우 (가천대학교 에너지IT학과) ;
  • 심귀보 (중앙대학교 전자전기공학부)
  • Received : 2017.01.04
  • Accepted : 2017.02.17
  • Published : 2017.02.25


The Convolutional Neural Network(CNN) can be divided into two stages: feature extraction and classification. The hyperparameters such as kernel size, number of channels, and stride in the feature extraction step affect the overall performance of CNN as well as determining the structure of CNN. In this paper, we propose a method to optimize the hyperparameter in CNN feature extraction stage using Parameter-Setting-Free Harmony Search (PSF-HS) algorithm. After setting the overall structure of CNN, hyperparameter was set as a variable and the hyperparameter was optimized by applying PSF-HS algorithm. The simulation was conducted using MATLAB, and CNN learned and tested using mnist data. We update the parameters for a total of 500 times, and it is confirmed that the structure with the highest accuracy among the CNN structures obtained by the proposed method classifies the mnist data with an accuracy of 99.28%.


Supported by : 한국연구재단


  1. S. K. Lee, K. E. Ko and K. B. Sim, "Study on Improvement of Convergence in Harmony Search Algorithms," Journal of Korean Institute of Intelligent Systems, Vol. 21, no. 3, pp. 401-406, 2011.
  2. Z. W. Geem and K. B. Sim, "Parameter-setting-free harmony search algorithm," Applied Mathematics and Computation, Vol. 217, no. 8, pp. 3881-3889, 2010.
  3. Y. LeCun, C. Cortes and C. J. Burges, MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann., 2010, [Accessed: December 25, 2016]
  4. J. S. Ren and L. Xu, "On vectorization of deep convolutional neural networks for vision tasks," arXiv preprint arXiv:1501.07338, 2015.
  5. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
  6. J. H. Yu and K. B. Sim, "Face Classification Using Cascade Facial Detection and Convolutional Neural Network," Journal of Korean Institute of Intelligent Systems, vol. 26, no. 1, pp. 70-75, 2016.
  7. A. Krizhevsky, I. Sutskever and G. E. Hinton, "Imagenet classification with deep convolutional neural networks". In Advances in neural information processing systems, pp. 1097-1105, 2012.
  8. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," In European conference on computer vision, pp. 818-833, 2014.
  9. W. Wang, J. Yang, J. Xiao, S. Li and D. Zhou, "Face Recognition Based on Deep Learning," Human Centered Computing, pp. 812-820, 2014.
  10. S. Ahn, "Deep Learning Architectures and Applications," Journal of Intelligence and Information Systems, vol. 22, no. 2, pp. 127-142, 2016.
  11. X. S. Yang and Z. W. Geem, Music-inspired harmony search algorithm: theory and applications, 2009.
  12. K. S. Lee and Z. W. Geem, "A new meta-heuristic algorithm for continuous engineering optimization: harmony search theory and practice," Computer methods in applied mechanics and engineering, vol. 194, no. 36, pp. 3902-3933, 2005.
  13. T. J. Lee, S. M. Park, K. E. Ko, W. K. Sung and K. B. Sim, "Implementation of unsupervised nonlinear classifier with binary harmony search algorithm," Journal of Korean Institute of Intelligent Systems, Vol. 23, no. 4, pp. 354-359, 2013.
  14. G. S. Choi, C. Yu, R. M. Jin, S. K. Yu and M. G. Chun, "Short-term water demand forecasting algorithm using AR model and MLP," Journal of Korean Institute of Intelligent Systems, vol. 19, no. 5, pp.713-719, 2009.
  15. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86 no. 11, pp. 2278-2324, 1998.