Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Neural Network based Video Coding in JVET

  • Received : 2022.10.17
  • Accepted : 2022.12.02
  • Published : 2022.12.20
Download PDF
⟨ Previous Next ⟩

Abstract

After the Versatile Video Coding (VVC)/H.266 standard was completed, the Joint Video Exploration Team (JVET) began to investigate new technologies that could significantly increase coding gain for the next generation video coding standard. One direction is to investigate signal processing based tools, while the other is to investigate Neural Network based technology. Neural Network based Video Coding (NNVC) has not been studied previously, and this is the first trial of such an approach in the standard group. After two years of research, JVET produced the first common software called Neural Compression Software (NCS) with two NN-based in-loop filtering tools at the 27th meeting and began to maintain NN-based technologies for the common experiment. The coding performances of the two filters in NCS-1.0 are shown to be 8.71% and 9.44% on average in a random access scenario, respectively. All the material related to NCS can be found in the repository of the JVET. In this paper, we provide a brief overview and review of the NNVC activity studied in JVET in order to provide trend and insight for the new direction of video coding standard.

Keywords

Acknowledgement

This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (IITP-2021-0-02067) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (NRF-2021R1F1A1060816).

References

  1. Versatile Video Coding, Recommendation ITU-T H.266 and ISO/IEC 23090-3 (VVC), ITU-T and ISO/IEC JTC 1, Jul. 2020.
  2. High Efficiency Video Coding, Recommendation ITU-T H.265 and ISO/IEC 23008-2 (HEVC), ITU-T and ISO/IEC JTC 1, Apr. 2013.
  3. S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, S. Wang, "Image and video compression with neural networks: A review." IEEE Transactions on Circuits and Systems for Video Technology, 2019. doi: https://doi.org/10.1109/TCSVT.2019.2910119
  4. A. Alshina, S. Liu, J. Pfaff, M. Wien, P. Wu, Y. Ye, "JVET AHG report: Neural-network-based video coding (AHG11)", JVET-T0011, Oct. 2020.
  5. L. Wang, S. Lin, X. Xu, S. Liu (Tencent), F. Galpin (InterDigital), "EE1-1.5: Neural network based in-loop filter with a single model", JVET-AA0088, Jul. 2022.
  6. Y. Li, K. Zhang, J. Li, L. Zhang (Bytedance), H. Wang, M. Coban, A.M. Kotra, M. Karczewicz (Qualcomm), F. Galpin (InterDigital), K. Andersson, J. Strom, D. Liu, R. Sjoberg (Ericsson), "EE1-1.6: Deep In-Loop Filter With Fixed Point Implementation", JVET-AA0111, Jul. 2022.
  7. S. Peng, C. Fang, D. Jiang, J. Lin, X. Zhang (Dahua), J. Nam, S. Yoo, J. Lim, S. Kim (LGE), "EE1-2.1: A CNN-based Super Resolution Method Combined with GOP Level Adaptive Resolution", JVET-AA0071, Jul. 2022. https://jvet-experts.org/doc_end_user/documents/27_Teleconference/wg11/JVET-AA0071-v2.zip
  8. J. Nam, S. Yoo, J. Lim, S. Kim (LGE), "EE1-2.1: RPR encoder with multiple scale factors", JVET-Z0065, Apr. 2022.
  9. S. Peng, D. Jiang, J. Lin, C. Fang, X. Zhang (Dahua), "AHG11: A CNN-based Super Resolution Method Combined with Existing RPR Functionality", JVET-Z0088, Apr. 2022.
  10. Y. He, B. Wang, E. Alshina, J. Sauer, "AHG11: A hybrid codec using E2E image coding combined with VVC video coding", JVET-AA0063, Jul. 2022.
  11. Qipu Qin, Cheolkon Jung, Zou Dan, Ming Li, "[AHG11 & AHG6] DOVC: Deep Omnidirectional Video Compression", JVET-X0043, Oct. 2021.
  12. Qipu Qin, Cheolkon Jung (Xidian University), Dan Zou, Ming Li (OPPO), "AHG11: Deep omnidirectional video compression in YUV domain", JVET-Y0051, Jan. 2022.
  13. E. Alshina, R.-L. Liao, S. Liu, A. Segall, "Common Test Conditions and evaluation procedures for neural network-based video coding technology", JVET-Z2016, Apr. 2022.
  14. G. Bjontegaard, "Improvement of BD-PSNR Model", ITU-T SG16/Q6 VCEG-AI11, Jul. 2008.
  15. VVC Reference Software. https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tags/
  16. Neural Compression Software (NCS). https://vcgit.hhi.fraunhofer.de/jvet-ahg-nnvc/VVCSoftware_VTM/-/tree/VTM-11.0_nnvc
  17. JVET Common Test Conditions for Neural Network-Based Video Coding Technology. https://vcgit.hhi.fraunhofer.de/jvet-ahg-nnvc/nnvc-ctc/-/tree/master