Journal of KIISE (정보과학회 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Hole-Filling Method for Depth-Image-Based Rendering for which Modified-Patch Matching is Used

개선된 패치 매칭을 이용한 깊이 영상 기반 렌더링의 홀 채움 방법

  • 조재형 (현대모비스 연구개발본부) ;
  • 송원석 (삼성전자 영상디스플레이사업부) ;
  • 최혁 (서울시립대학교 컴퓨터과학부)
  • Received : 2016.11.09
  • Accepted : 2016.11.28
  • Published : 2017.02.15
⟨ Previous Next ⟩

Abstract

Depth-image-based rendering is a technique that can be applied in a variety of 3D-display systems. It generates the images that have been captured from virtual viewpoints by using a depth map. However, disoccluded hole-filling problems remain a challenging issue, as a newly exposed area appears in the virtual view. Image inpainting is a popular approach for the filling of the hole region. This paper presents a robust hole-filling method that reduces the error and generates a high quality-virtual view. First, the adaptive-patch size is decided using the color and depth information. Also, a partial filling method for which the patch similarity is used is proposed. These efforts reduce the error occurrence and the propagation. The experiment results show that the proposed method synthesizes the virtual view with a higher visual comfort compared with the existing methods.

깊이 영상 기반 렌더링은 깊이 정보를 활용하여 가상 시점의 영상을 생성하는 기술로 다양한 3차원 영상시스템에서 필요로 하는 기술이다. 깊이 영상 기반 렌더링에서 가장 어려운 과제는 가상 시점 영상에서 새롭게 드러나는 부분을 채우는 과정이다. 영상 인페인팅은 이 과정에서 보편적으로 활용되는 방법이다. 본 논문에서는 홀을 채우는 과정에서 발생하는 오류를 줄이고 자연스럽게 채우는 방법을 제안한다. 먼저 색상 영상의 정보와 깊이 정보를 활용하여 지역적으로 적응적 패치 크기를 선택하도록 하였다. 또한 패치 간 유사도에 따라 홀을 채우는 방법을 한 번에 채우는 경우와 부분적으로 채우는 경우로 구분하였다. 이를 통해 오류의 발생을 줄이고 깊이 영상 기반 렌더링에서 가장 큰 문제가 되는 오류의 전파를 억제하였다. 실험을 통해 제안한 방법이 기존의 방법보다 시각적으로 자연스러운 가상 시점 영상을 생성하는 것을 확인하였다.

Keywords

Acknowledgement

Supported by : 한국연구재단

References

  1. C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," SPIE In Electronic Imaging., pp. 93-104, 2004.
  2. L. Zhang, and W. J. Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Trans. Broadcasting, Vol. 51, No. 2, pp. 191-199, 2005. https://doi.org/10.1109/TBC.2005.846190
  3. W. Chen, Y. Chang, S. Lin, L. Ding, and L. Chen, "Efficient depth image based rendering with edge dependent depth filter and interpolation," IEEE International Conference on Multimedia and Expo., 2005.
  4. A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. Image Processing, Vol. 13, No. 9, pp. 1200-1212, 2004. https://doi.org/10.1109/TIP.2004.833105
  5. B. S. Kim, "An exemplar-based image inpainting method using structure matrix," Journal of KIISE: Software and Applications, Vol. 39, No. 7, pp. 583-592, 2012. (in Korean)
  6. C. Zhu, and S. Li, "Depth image based view synthesis: New insights and perspectives on hole generation and filling," IEEE Trans. Broadcasting, Vol. 62, No. 5, pp. 82-93, 2016. https://doi.org/10.1109/TBC.2015.2475697
  7. Y. Mao, G. Cheung, and Y. Ji, "On Constructing z-dimensional DIBR-Synthesized Images," IEEE Trans. Multimedia, Vol. 18, No. 8, pp. 1453-1468, 2016. https://doi.org/10.1109/TMM.2016.2573142
  8. K. J. Oh, S. Yea, and Y. S. Ho, "Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-d video," IEEE Picture Coding Symposium, pp. 1-4, 2009.
  9. L. Azzari, F. Battisti, A. Gotchev, M. Carli, and K. Egiazarian, "A modified non-local mean inpainting technique for occlusion filling in depth-image-based rendering," SPIE In Electronic Imaging, pp. 78631C-78631C, 2011.
  10. I. Daribo, and B. Pesquet-Popescu, "Depth-aided image inpainting for novel view synthesis," IEEE International Workshop on Multimedia Signal Processing, pp. 167-170, 2010.
  11. H. Zhou, and J. Zheng, "Adaptive patch size determination for patch-based image completion," IEEE International Conference on Image Processing, pp. 421-424, 2010.
  12. X. Lu, F. Wei, and F. Chen, "Foreground-objectprotected depth map smoothing for DIBR," IEEE International Conference on Multimedia and Expo., pp. 339-343, 2012.
  13. "Microsoft Research 3D Video Dataset," [Online]. Available: https://www.microsoft.com/en-us/download/
  14. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Trans. Image Processing, Vol. 13, No. 2, pp. 600-612, 2004. https://doi.org/10.1109/TIP.2003.819861