Journal of KIISE (정보과학회 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

A Personal Video Event Classification Method based on Multi-Modalities by DNN-Learning

DNN 학습을 이용한 퍼스널 비디오 시퀀스의 멀티 모달 기반 이벤트 분류 방법

  • 이유진 (서강대학교 컴퓨터공학과) ;
  • 낭종호 (서강대학교 컴퓨터공학과)
  • Received : 2016.05.17
  • Accepted : 2016.08.05
  • Published : 2016.11.15
⟨ Previous Next ⟩

Abstract

In recent years, personal videos have seen a tremendous growth due to the substantial increase in the use of smart devices and networking services in which users create and share video content easily without many restrictions. However, taking both into account would significantly improve event detection performance because videos generally have multiple modalities and the frame data in video varies at different time points. This paper proposes an event detection method. In this method, high-level features are first extracted from multiple modalities in the videos, and the features are rearranged according to time sequence. Then the association of the modalities is learned by means of DNN to produce a personal video event detector. In our proposed method, audio and image data are first synchronized and then extracted. Then, the result is input into GoogLeNet as well as Multi-Layer Perceptron (MLP) to extract high-level features. The results are then re-arranged in time sequence, and every video is processed to extract one feature each for training by means of DNN.

최근 스마트 기기의 보급으로 자유롭게 비디오 컨텐츠를 생성하고 이를 빠르고 편리하게 공유할 수 있는 네트워크 환경이 갖추어지면서, 퍼스널 비디오가 급증하고 있다. 그러나, 퍼스널 비디오는 비디오라는 특성 상 멀티 모달리티로 구성되어 있으면서 데이터가 시간의 흐름에 따라 변화하기 때문에 이벤트 분류를 할 때 이에 대한 고려가 필요하다. 본 논문에서는 비디오 내의 멀티 모달리티들로부터 고수준의 특징을 추출하여 시간 순으로 재배열한 것을 바탕으로 모달리티 사이의 연관관계를 Deep Neural Network(DNN)으로 학습하여 퍼스널 비디오 이벤트를 분류하는 방법을 제안한다. 제안하는 방법은 비디오에 내포된 이미지와 오디오를 시간적으로 동기화하여 추출한 후 GoogLeNet과 Multi-Layer Perceptron(MLP)을 이용하여 각각 고수준 정보를 추출한다. 그리고 이들을 비디오에 표현된 시간순으로 재 배열하여 비디오 한 편당 하나의 특징으로 재 생성하고 이를 바탕으로 학습한 DNN을 이용하여 퍼스널 비디오 이벤트를 분류한다.

Keywords

Acknowledgement

Grant : 퍼스널 미디어가 연결공유결합하여 재구성 가능케 하는 복합 모달리티 기반 미디어 응용 프레임워크 개발

Supported by : 정보통신기술진흥센터

References

  1. J.-H. Shin, S.-K. Baek and P.-K. Kim, "Video Event Detection according to Generating of Semantic Unit based on Moving Object," Journal of Korea Multimedia Society, Vol. 11, No. 2, pp. 143-152, 2008.
  2. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar and L. Fei-Fei, "Large-Scale Video Classification with Convolutional Neural Networks," Proc. of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Vol. 37, pp. 448-456, 2014.
  3. S. Yu et al., "CMU-Informedia@ TRECVID 2014 Multimedia Event Detection (MED)," Proc. of the 2014 TRECVID Video Retrieval Evaluation Workshop, 2014.
  4. J. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga and G. Toderici, "Beyond Short Snippets: Deep Networks for Video Classification," Proc. of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4694-4702, 2015.
  5. Z. Wu, Y.-G. Jiang, J. Wang, J. Pu and X. Xue, "Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification," Proc. of the 22nd ACM International Conference on Multimedia, pp. 167-176, 2014.
  6. C. Szegedy et al., "Going Deeper with Convolutions," Proc. of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015.
  7. F. Rosenblatt, Principles of Neurodynamics, SpartanBook, 1962.
  8. Soomro, A. R. Zamir and M. Shah, UCF101: A Dataset of 101 Human Action Classes From Videos in The Wild, CRCV-TR-12-01, 2012.
  9. A. Krizhevsky, I. Sutskever and G. E. Hinton, "ImageNet Classificattion with Deep Convolutional Neural Networks," Proc. of Neural Information Processing Systems 2012, 2012.
  10. Y. LeCunn, L. Bottou, Y. Bengio and P. Haffiner, "Gradient-Based Learning Applied to Document Recognition," Proc. of IEEE, Vol. 86, No. 11, pp. 2278-2324, 1998. https://doi.org/10.1109/5.726791
  11. S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, Vol. 9, No. 8, pp. 1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
  12. K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Visual Recognition," Proc. of International Conference on Learning Representations 2014, 2014.
  13. B. Zhu, W. Li and X. Xue, "A Novel Audio Fingerprinting Method Robust to Time Scale Modification and Pitch Shifting," Proc. of the 18th ACM International Conference on Multimedia, pp. 987-990, 2010.
  14. M. Sahidullah and Goutam Saha, "Design, Analysis and Experimental Evaluation of Block based Transformation in MFCC Computation for Speaker Recognition," Journal of Speech Communication, Vol. 54, No. 4, pp. 543-565, 2012. https://doi.org/10.1016/j.specom.2011.11.004
  15. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama and T. Darrell, "Caffe: Convolutional Architecture for Fast Feature Embedding," Proc. of the 22nd ACM International Conference on Multimedia, pp. 675-678, 2014.
  16. LISA Lab, Theano, http://deeplearning.net/software/theano/, 2015.
  17. D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, "Learning Spatiotemporal Features with 3D Convolutional Networks," Proc. of International Conference on Computer Vision 2015, pp. 435-442, 2015.
  18. N. Srivastava, E. Mansimov and R. Salakhutdinov, "Unsupervised Learning of Video Representations using LSTMs," Proc. of the 32nd International Conference on Machine Learning, pp. 843-852, 2015.
  19. H. Ye, Z. Wu, R.-W. Zhao, X. Wang, Y.-G. Jiang and X. Xue, "Evaluating Two-Stream CNN for Video Classification," Proc. of the 5th ACM International Conference on Multimedia Retrieval, pp. 435-442, 2015.