Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Efficient Memory Update Module for Video Object Segmentation

동영상 물체 분할을 위한 효율적인 메모리 업데이트 모듈

  • Jo, Junho (Department of ECE, INMC, Seoul National University) ;
  • Cho, Nam Ik (Department of ECE, INMC, Seoul National University)
  • 조준호 (서울대학교 전기정보공학부) ;
  • 조남익 (서울대학교 전기정보공학부)
  • Received : 2022.05.20
  • Accepted : 2022.07.20
  • Published : 2022.07.30
Download PDF
⟨ Previous Next ⟩

Abstract

Most deep learning-based video object segmentation methods perform the segmentation with past prediction information stored in external memory. In general, the more past information is stored in the memory, the better results can be obtained by accumulating evidence for various changes in the objects of interest. However, all information cannot be stored in the memory due to hardware limitations, resulting in performance degradation. In this paper, we propose a method of storing new information in the external memory without additional memory allocation. Specifically, after calculating the attention score between the existing memory and the information to be newly stored, new information is added to the corresponding memory according to each score. In this way, the method works robustly because the attention mechanism reflects the object changes well without using additional memory. In addition, the update rate is adaptively determined according to the accumulated number of matches in the memory so that the frequently updated samples store more information to maintain reliable information.

최근 대부분의 딥러닝 기반 동영상 물체 분할 방법들에서는 외부 메모리에 과거 예측 정보를 저장한 상태에서 알고리즘 수행을 하며, 일반적으로 메모리에 많은 과거 정보를 저장할수록 관심 물체의 다양한 변화에 대한 근거들이 축적되어 좋은 결과를 얻을 수 있다. 하지만 하드웨어의 제한으로 인해 메모리에 모든 정보를 저장할 수 없어 이에 따른 성능 하락이 발생한다. 본 논문에서는 저장되지 않는 정보들을 기존의 메모리에 추가적인 메모리 할당 없이 저장하는 방법을 제안한다. 구체적으로, 기존 메모리와 새로 저장할 정보들과의 어텐션 점수를 계산한 후에, 각 점수에 따라 해당 메모리에 새 정보를 더한다. 이 방법으로 물체 형체의 변화에 대한 정보가 반영되어 물체 변화에 대한 강인성이 높아져서 분할 성능이 유지됨을 확인할 수 있었다. 또한, 메모리의 누적 매칭 횟수에 따라 적응적으로 업데이트 비율을 결정하여, 업데이트가 많이 되는 샘플들은 과거의 정보를 더 기억하여 신뢰성 있는 정보를 유지할 수 있게 하였다.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (2021R1A2C2007220).

References

  1. Oh, S. W., Lee, J. Y., Xu, N., & Kim, S. J., "Video object segmentation using space-time memory networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. doi: https://doi.org/10.1109/ICCV.2019.00932
  2. Li, Yu, Zhuoran Shen, and Ying Shan., "Fast video object segmentation using the global context module." European Conference on Computer Vision. Springer, Cham, 2020. doi: https://doi.org/10.1007/978-3-030-58607-2_43
  3. Liang, Y., Li, X., Jafari, N., & Chen, J., "Video object segmentation with adaptive feature bank and uncertain-region refinement." Advances in Neural Information Processing Systems 33: 3430-3441., 2020.
  4. Pont-Tuset, J., Perazzi, F., Caelles, S., Arbelaez, P., Sorkine-Hornung, A., & Van Gool, L, "The 2017 davis challenge on video object segmentation." arXiv preprint arXiv:1704.00675, 2017.
  5. Ning Xu, Linjie Yang, Dingcheng Yue, Jianchao Yang, Brian Price, Jimei Yang, Scott Cohen, Yuchen Fan, Yuchen Liang, and Thomas Huang., "Youtube-vos: Sequence-to-sequence video object segmentation." In European Conference on Computer Vision (ECCV), 2018. doi: https://doi.org/10.1007/978-3-030-01228-1_36
  6. Yao, R., Lin, G., Xia, S., Zhao, J., & Zhou, Y., "Video object segmentation and tracking: A survey." ACM Transactions on Intelligent Systems and Technology (TIST) 11.4 ,1-47p, 2020. doi: http://dx.doi.org/10.1145/3391743
  7. Wang, H., Jiang, X., Ren, H., Hu, Y., & Bai, S., "Swiftnet: Real-time video object segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. doi: https://doi.org/10.1109/CVPR46437.2021.00135
  8. Hu, Yuan-Ting, Jia-Bin Huang, and Alexander G. Schwing. "Video-match: Matching based video object segmentation." Proceedings of the European conference on computer vision (ECCV). 2018. doi: https://doi.org/10.1007/978-3-030-01237-3_4
  9. He, K., Zhang, X., Ren, S., & Sun, J., "Deep residual learning for image recognition.", Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770-778, 2016. doi: https://doi.org/10.1109/CVPR.2016.90
  10. T. -Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature Pyramid Networks for Object Detection.", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 936-944, 2017. doi: https://doi.org/10.1109/CVPR.2017.106