Journal of Korea Multimedia Society (한국멀티미디어학회논문지)

Korea Multimedia Society (한국멀티미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Comparison of Image Classification Performance by Activation Functions in Convolutional Neural Networks

컨벌루션 신경망에서 활성 함수가 미치는 영상 분류 성능 비교

  • Received : 2018.07.25
  • Accepted : 2018.08.29
  • Published : 2018.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, computer vision application is increasing by using CNN which is one of the deep learning algorithms. However, CNN does not provide perfect classification performance due to gradient vanishing problem. Most of CNN algorithms use an activation function called ReLU to mitigate the gradient vanishing problem. In this study, four activation functions that can replace ReLU were applied to four different structural networks. Experimental results show that ReLU has the lowest performance in accuracy, loss rate, and speed of initial learning convergence from 20 experiments. It is concluded that the optimal activation function varied from network to network but the four activation functions were higher than ReLU.

Keywords

References

  1. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, et al., "Mastering the Game of Go without Human Knowledge," Nature, Vol. 550, pp. 354-359, 2017. https://doi.org/10.1038/nature24270
  2. L. Yann, B. Yoshua, and H. Geoffrey, “Deep leaning,” Nature, Vol. 521, No. 7553, pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  3. H. Huang, J. Joseph, M. Huang, and T. Margolina, "Automated Detection and Identification of Blue and Fin Whale Foraging Calls by Combining Pattern Recognition and Machine Learning Techniques," Proceeding of OCEANS 2016 MTS/IEEE Monterey, pp. 1-7, 2016.
  4. N. Akhtar and A. Mian, "Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey," arXiv preprint, arXiv:1801.00553, 2018.
  5. R. Olga, "ImageNet Large ScaleVisual Recognition Challenge," International Journal of Computer Vision, Vol. 115, No. 3, pp. 211-252, 2015. https://doi.org/10.1007/s11263-015-0816-y
  6. K. Alex, S. Ilya, and H. Geoffrey, "ImageNet Classification with Deep Convolutional Neural Networks," Proceeding of Advances in Neural Information Processing System, pp. 1097-1105, 2012.
  7. V. Nair and G. Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," Proceedings of the 27th International Conference on Machine Learning, pp. 807-814, 2010.
  8. M. Lin, Q. Chen, and S. Yan, "Network In Network," arXiv, arXiv:1312.4400v3, 2014.
  9. S. loffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," arXiv preprint, arXiv:1502.03167v3, 2015.
  10. S. Zagoruyko and N. Komodakis, "Wide Residual Networks," arXiv preprint, arXiv: 1605.07146, 2016.
  11. G. Huang, Z. Liu, L.V.D. Maaten, and K.Q. Weinberger, "Densely Connected Convolutional Networks," Proceeding of 2017 IEEE Conference on Computer Vision and P attern Recognition, pp. 2261-2269, 2017.
  12. J. Yangqing, "Caffe: An Open Source Convolutional Architecture for Fast Feature Embedding," Proceeding of the 22nd ACM International Conference on Multimedia, pp. 675-678, 2014.
  13. A.L. Maas, A.Y. Hannun, and A.Y. Ng, "Rectifier Nonlinearities Improve Neural Network Acoustic Models," Proceeding of International Conference on Machine Learning, 2013.
  14. K. He, X. Zhang, S. Ren, and J. Sun, "Delving Deep Into Rectifiers: Surpassing HumanLevel Performance On Imagenet Classification," Proceeding of International Conference on Computer Vision, pp. 1026-1034, 2015.
  15. B. Xu, N. Wang, T. Chen, and M. Li, "Empirical Evaluation of Rectified Activations in Convolutional Network," arXiv, arXiv:1505.00853, 2015.
  16. D.-A. Clevert, T. Unterthiner, and S. Hochreiter, "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)," arXiv, arXiv:1511.07289, 2015.
  17. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. reed, D. Anguelov, et al., "Going Deeper with Convolutions," Proceedings of the IEEE Conference on Computer Vision and P attern Recognition, pp. 1-9, 2015.
  18. Y. Jeong, l. Ansari, J. Shim and J. Lee, "A Car Plate Area Detection System Using Deep Convolution Neural Network," Journal of Korea Multimedia Society, Vol. 20, No. 8, pp. 1166-1174, 2017. https://doi.org/10.9717/KMMS.2017.20.8.1166
  19. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, "What is the Best Multi-stage Architecture for Object Recognition?," Proceeding of IEEE 12th International Conference on Computer Vision, pp. 2146-2153, 2009.
  20. V. Nair and G. Hinton, "Rectified lInear Units Improve Restricted Boltzmann Machines," 2010 27th International Conference on Machine Learning, pp. 807-814, 2010.
  21. X. Glorot, A. Bordes, and Y. Bengio, "Deep Sparse Rectifier Neural Networks," Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR, Vol. 15, pp. 315-323, 2011.
  22. I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the Importance of Initialization and Momentum in Deep Learning," Proceedings of the 30th International Conference on Machine Learning, Vol. 28, pp. 1139-1147, 2013.