The Journal of Korean Institute of Communications and Information Sciences (한국통신학회논문지)

The Korean Institute of Commucations and Information Sciences (한국통신학회)
권호 표지
DOI QR코드

DOI QR Code

Real-Time Virtual-View Image Synthesis Algorithm Using Kinect Camera

키넥트 카메라를 이용한 실시간 가상 시점 영상 생성 기법

  • 이규철 (광운대학교 전자공학과 디지털미디어 연구실) ;
  • 유지상 (광운대학교 전자공학과 디지털미디어 연구실)
  • Received : 2013.02.14
  • Accepted : 2013.05.01
  • Published : 2013.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

Kinect released by Microsoft in November 2010 is a motion sensing camera in xbox360 and gives depth and color images. However, Kinect camera also generates holes and noise around object boundaries in the obtained images because it uses infrared pattern. Also, boundary flickering phenomenon occurs. Therefore, we propose a real-time virtual-view video synthesis algorithm which results in a high-quality virtual view by solving these problems. In the proposed algorithm, holes around the boundary are filled by using the joint bilateral filter. Color image is converted into intensity image and then flickering pixels are searched by analyzing the variation of intensity and depth images. Finally, boundary flickering phenomenon can be reduced by converting values of flickering pixels into the maximum pixel value of a previous depth image and virtual views are generated by applying 3D warping technique. Holes existing on regions that are not part of occlusion region are also filled with a center pixel value of the highest reliability block after the final block reliability is calculated by using a block based gradient searching algorithm with block reliability. The experimental results show that the proposed algorithm generated the virtual view image in real-time.

키넥트 카메라는 마이크로소프트사에서 2010년 11월에 출시한 xbox360의 움직임 감지 카메라로 깊이 영상과 색상 영상을 획득할 수 있다. 하지만 적외선 패턴을 이용한 깊이 영상의 획득 방법의 한계로 인해 객체의 경계 주변으로 홀(hole) 및 잡음이 생기고 영상으로 재생 시 경계 주변에서 흔들림(flickering) 현상이 발생한다. 본 논문에서는 이러한 흔들림 현상을 보정하여 화질이 좋은 가상 시점 영상을 실시간으로 생성하는 기법을 제안한다. 제안하는 기법에서는 먼저 결합형 양방향 필터를 이용하여 경계 주변의 홀을 채운다. 경계 주변의 흔들림 현상은 화소를 탐색하여 처리하는 기법을 적용하여 보정한다. 향상된 깊이 영상과 색상 영상에 3D 워핑(3D warping) 기법을 적용하여 가상 시점 영상을 획득한다. 획득된 영상에서 가려짐 영역(occlusion region)으로 인하여 생기는 홀은 블록기반의 기울기 탐색 기법과 블록의 신뢰도를 이용하여 채우게 된다. 실험을 통해 제안하는 시스템이 가상 시점 영상을 실시간으로 합성하는 것을 확인하였다.

Keywords

References

  1. Retrevo Corporation, Could low interest in 3DTV hurt the TV business?, Retrieved Nov., 2011, from http://www.retrevo.com/content/node/1915.
  2. G. M. Um, G. H. Cheong, W. S. Cheong, and N. H. Hur, "Technical development and standardization trends of multi-view 3D and free-viewpoint video," IEEK Mag., vol. 38, no. 2, pp. 18-23, Feb. 2011.
  3. T. J. Kim and J. S. Yoo, "Hierarchical stereo matching with color information," J. KICS, vol. 34, no. 3, pp. 279-287, Mar. 2009.
  4. T. Leyvand, C. Meekhof, Y. C. Wei, J. Sun, and B. Guo, "Kinect identity: technology and experience," Computer (IEEE Comput. Soc.), vol. 44, no. 4, pp. 94-96, Apr. 2011.
  5. Y. S. Park, S. M. Yun, and C. S. Won, "Hole filling for kinect depth image according to the causes of the holes," in Proc. Conf. Korean Soc. Broadcast Eng. (KOSBE), pp. 75-80, Jeju Island, Korea, July 2012.
  6. C. J. Park, J. H. Ko, and E. S. Kim, "A new intermediate view reconstruction scheme based-on stereo image rectification algorithm," J. KICS, vol. 29, no. 5C, pp. 632-641, May. 2004.
  7. M. S. Ko and J. S. Yoo, "Boundary noise removal and hole filling algorithm for virtual viewpoint image generation," J. KICS, vol. 37, no. 8, pp. 679-688, Aug. 2012. https://doi.org/10.7840/kics.2012.37A.8.679
  8. L. Zhao and H. Wang, "Image denoising using trivariate shrinkage filter in the wavelet domain and joint bilateral filter in the spatial Domain," IEEE Trans. Image Process., vol. 18, no. 10, pp. 2364-2369, Oct. 2009. https://doi.org/10.1109/TIP.2009.2026685
  9. A. Telea, "An image inpainting technique based on the fast marching method," J. Graphics Tools, vol. 9, no. 1, pp. 25-36, Dec. 2004.
  10. A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar-based image in-painting," IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200-1212, Sept. 2004. https://doi.org/10.1109/TIP.2004.833105
  11. G. C. Lee, Y. H. Seo, and J. S. Yoo, "GPGPU-based multiview synthesis using kinect depth image," in Proc. Conf. KICS, pp. 152-154, Yongpyong, Korea, Jan. 2012.
  12. Y. Mori, N. Fukushima, T. Yendoa, T. Fujii, and M. Tanimotoa, "View generation with 3D warping using depth information for FTV," Signal Processing : Image Communication (ELSEVIER), vol. 24, no. 1-2, pp. 65-72, Jan. 2009. https://doi.org/10.1016/j.image.2008.10.013
  13. Y. J. Kim, S. H. Lee, and J. I. Park, "A high-quality occlusion filling method using image inpainting," J. Korean Soc. Broadcast Eng. (KOSBE), vol. 15, no. 1, pp. 3-13, Jan. 2010. https://doi.org/10.5909/JBE.2010.15.1.003
  14. S. B. Lee and Y. S. Ho, "Real time eye contact system using a kinect depth camera for realistic telepresence," J. KICS, vol. 37, no. 4, pp. 277-282, Apr. 2012. https://doi.org/10.7840/KICS.2012.37C.4.277
  15. W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, "Smoothing depth maps for improved stereoscopic image quality," in Proc. SPIE Conf. Three-Dimensional TV, Video, Display III, vol. 5599, pp. 162-172, Philadelphia, U.S.A., Oct. 2004.
  16. T. Y. Kim, Y. G. Jeon, and J. C. Jeong, "Adaptive linear interpolation using the new distance weight and local patterns," J. KICS, vol. 31, no. 12C, pp. 1184-1193, Dec. 2006.
  17. G. C. Lee and J. S. Yoo, "Kinect depth map enhancement using boundary flickering compensation," in Proc. Conf. Korean Soc. Broadcast Eng. (KOSBE), pp. 25-28, Seoul, Korea, Nov. 2012.
  18. T. J. Kim, E. Y. Chang, N. H. Hur, J. W. Kim, and J. S. Yoo, "Virtual viewpoint image synthesis algorithm using multi-view geometry," J. KICS, vol. 34, no. 12, pp. 1154-1166, Dec. 2009.
  19. J. H. Park and C. G. Song, "Effective shadow removal from aerial image of golf course to extract components," J. Korean Inst. of Inform. Sci. Eng. (KIISE), vol. 39, no. 7, pp. 577-582, July 2012
  20. N. E. Yang, Y. G. Kim, and R. H. Hong, "Depth hole filling using the depth distribution of neighboring regions of depth holes in the kinect sensor," in Proc. IEEE Int. Conf. Signal Process., Commun. Comput. (ICSPCC), pp. 658-661, Hong Kong, China, Aug. 2012
  21. S.-B. Lee and Y.-S. Ho, "Real-time stereo view generation using kinect depth camera," in Proc. Asia-Pacific Signal Inform. Process. Assoc. Annu. Summit Conf. (APSIPA ASC) 2011, pp. 1-4, Xi'an, China, Oct. 2011.