그림 1. 개인방송 플랫폼 구성 (좌: Twitch[1], 우: Kakao TV[2]) Fig. 1. Examples of live streaming platforms (left: Twitch[1], right: Kakao TV[2])
그림 2. 단일 시구간 모델 STIM Fig. 2. Single Time Interval Model(STIM)
그림 3. 다중 시구간 모델 MTIM Fig. 3. Multiple Time Interval Model(MTIM)
그림 4. 다중 데이터를 이용하는 하이라이트 예측 모델. (a) M-STIM, (b) M-MTIM Fig. 4. Highlight prediction models using multimodal data. (a) M-STIM, (b) M-MTIM
그림 5. e스포츠 영상에 대한 모델별 실험 결과 (파란 실선: 하이라이트 구간 위치, 빨간 점선: 하이라이트 스코어) Fig. 5. Experiment results on an e-Sports test video (blue: highlight locations, red: highlight score)
그림 6. 야구 영상에 대한 모델별 실험 결과 (2000~4000초, 파란 실선: 하이라이트 구간 위치, 빨간 점선: 하이라이트 스코어) Fig. 6. Experiment results on a baseball video (2000~4000sec, blue: locations of highlights, red: highlight score)
표 1. e스포츠와 야구경기 데이터 요약 정보 Table 1. Summary of e-Sports and baseball datasets
표 2. e스포츠 데이터 7개에 대한 실험 결과 Table 2. Experiment results on e-Sports data
표 3. 야구 데이터 5개에 대한 실험 결과 Table 3. Experiment results on baseball data
표 4. 야구 데이터에서 긴 구간의 길이에 따른 F1 점수 Table 4. F1 scores evaluated for various long-term intervals
Algorithm 1. STIM
Algorithm 2. MTIM
Algorithm 3. M-STIM
Algorithm 4. M-MTIM
참고문헌
- Twitch, https://www.twitch.tv/ (accessed Mar. 08, 2019).
- Kakao TV, https://tv.kakao.com/ (accessed Mar. 08, 2019).
- M. Sun, A. Farhadi, and S. Seitz, "Ranking Domain -specific Highlights by Analyzing Edited Videos," European Conference on Computer Vision, Zurich, Switzerland, pp. 708-802, 2014, doi:10.1007/978-3-319-10590-1_51.
- H. Tang, V. Kwatra, ME. Sargin, and U. Gargi, "Detecting highlights in sports videos: Cricket as a test case," IEEE International Conference on Multimedia and Expo, Barcelona, Spain, pp. 1-6, 2011, doi:10.1109/ICME.2011.6012139.
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," The IEEE Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, pp. 1-9, 2015, doi: 10.1109/CVPR.2015.7298594 .
- K. Zhang, WL. Chao, F. Sha, and K. Grauman, "Video Summarization with Long Short-term Memory," European Conference on Computer Vision, Amsterdam, Netherlands, pp. 766-782, 2016, doi:10.1007/978-3-319-46478-7_47.
- Z. Xiong, R. Radhakrishnan, A. Divakaran, and TS. Huang, "Highlights extraction from sports video based on an audio-visual marker detection framework", IEEE International Conference on Multimedia and Expo, Amsterdam, Netherlands, pp. 29-32, 2005, doi:10.1109/ICME.2005.1521352.
- LC. Hsieh, CW. Lee, TH. Chiu, and W. Hsu, "Live semantic sport highlight detection based on analyzing tweets of twitter," IEEE International Conference on Multimedia and Expo, Melbourne, Australia, pp. 949-954, 2012, doi:10.1109/ICME.2012.135.
- J. Li, Z. Liao, C. Zhang, and J. Wang, "Event Detection on Online Videos using Crowdsourced Time-Sync Comment," International Conference on Cloud Computing and Big Data, Macau, China, pp. 52-57, 2016, doi:10.1109/CCBD.2016.021.
- Q. Ping, C. Chen, "Video Highlights Detection and Summarization with Lag-Calibration based on Concept-Emotion Mapping of Crowd-sourced Time-Sync Comments," Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 1-11, 2017, doi:10.18653/v1/W17-4501.
- E. Kim, G. Lee, "Highlight Detection in Personal Broadcasting by Analysing Chat Traffic : Game Contests as a Test Case," Journal of Broadcast Engineering, Vol.23, No.2, pp.218-226, 2018, doi:http://dx.doi.org/10.5909/JBE.2018.23.2.218.
- CY. Fu, J. Lee, M. Bansal, and AC. Berg, "Video Highlight Prediction Using Audience Chat Reactions," Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 972-978, 2017.
- A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, "Bag of Tricks for Efficient Text Classification," European Chapter of the Association for Computational Linguistics, Valencia, Spain, pp. 427-431, 2016, doi:10.18653/v1/E17-2068.
- S. Davis, P.Mermelstein, "Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences," IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol.28, No.4, pp.357-366, 1980, doi:https://doi.org/10.1109/tassp.1980.1163420.
- T. Mikolov, K. Chen, G. Corrado, and J. Dean. "Efficient Estimation of Word Representations in Vector Space," Journal of Biomedical Science and Engineering, Vol.9, No.1, pp.7-16 2016 https://doi.org/10.4236/jbise.2016.91002
- S. Hochreiter, J. Schmidhuber, "Long short-Term Memory," Neural Computation, Vol.9, No.8, pp.1735-1780, 1997, doi:10.1162/neco.1997.9.8.1735.