Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Stereo-To-Multiview Conversion System Using FPGA and GPU Device

FPGA와 GPU를 이용한 스테레오/다시점 변환 시스템

  • Shin, Hong-Chang (Electronics and Telecommunications Research Institute) ;
  • Lee, Jinwhan (Electronics and Telecommunications Research Institute) ;
  • Lee, Gwangsoon (Electronics and Telecommunications Research Institute) ;
  • Hur, Namho (Electronics and Telecommunications Research Institute)
  • Received : 2014.07.21
  • Accepted : 2014.09.25
  • Published : 2014.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we introduce a real-time stereo-to-multiview conversion system using FPGA and GPU. The system is based on two different devices so that it consists of two major blocks. The first block is a disparity estimation block that is implemented on FPGA. In this block, each disparity map of stereoscopic video is estimated by DP(dynamic programming)-based stereo matching. And then the estimated disparity maps are refined by post-processing. The refined disparity map is transferred to the GPU device through USB 3.0 and PCI-express interfaces. Stereoscopic video is also transferred to the GPU device. These data are used to render arbitrary number of virtual views in next block. In the second block, disparity-based view interpolation is performed to generate virtual multi-view video. As a final step, all generated views have to be re-arranged into a single image at full resolution for presenting on the target autostereoscopic 3D display. All these steps of the second block are performed in parallel on the GPU device.

본 논문에서는 FPGA와 GPU를 이용한 실시간 스테레오 다시점 변환 시스템을 소개한다. 해당 시스템은 이종의 연산장치를 이용하며 그에 따라 크게 두 부분으로 나뉜다. 첫 번째 부분은 변이 추출 부분으로서 실시간 계산을 위해 FPGA기반으로 구현되었다. 기본적으로 DP(Dynamic programming) 기반의 스테레오 정합 방법을 통해 초기 변이 영상이 계산되며, 후처리를 통해 개선된다. 개선된 변이 영상은 USB3.0과 PCI-express를 통해 GPU 장치로 전송된다. 스테레오 입력 영상이 GPU장치로도 전송되면, 변이 영상의 변이 값을 이용하여 중간 시점에서의 영상을 합성한다. 생성된 시점 영상들은 무안경 다시점 3차원 디스플레이의 특성에 맞게 하나의 영상으로 화소 또는 부분화소 단위로 재배치되는 시점 다중화 과정을 거쳐 최종적으로 4K 무안경 다시점 디스플레이에 실시간으로 재생된다. 스테레오 정합을 제외한 나머지 연산은 모두 GPU에서 병렬처리된다

Keywords

References

  1. F. Zilly, C. Riechert, M. Muller, P. Eisert, T. Sikora, and Peter Kauff, "Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline," Journal of Visual Communication and Image Representation, 2013.
  2. M. Muller, F. Zilly, P. Kauff, "Adaptive cross-trilateral depth map filtering," in 3DTV-Conference, pp. 1-4, June 2010.
  3. S. Jin, J. Cho, X. D. Pham, K. M. Lee, S.-K. Park, M. Kim, and J.W.Jeon, "FPGA design and implementation of a real-time stereo vision system," IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 1, pp. 15 -26, 2010. https://doi.org/10.1109/TCSVT.2009.2026831
  4. L. Zhang, K. Zhang, T. S. Chang, G. Lafruit, G. K. Kuzmanov, and D. Verkest, "Real-time high-definition stereo matching on FPGA," Proceedings of the 19th ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 55-64, 2011.
  5. H.-C. Shin, G.-M. Um, C. Kim, W.-S. Cheong, and N. Hur, "Autostereoscopic 3D video generation from sterescopic vidoes using FPGA and GPU," Proceedings of IEEE International Conference on 3D Imaging(IC3D), 2012.
  6. K. Pauwels, M. Tomasi, J. Dias, E. Ros, and M. M. Van Hulle, "A Comparison of FPGA and GPU for Real-Time Phase-Based Optical Flow, Stereo, and Local Image Features," IEEE Transaction on Computers, vol. 61, no. 7, pp. 999-1012, 2012. https://doi.org/10.1109/TC.2011.120
  7. Y. Oh and H. Jeong, "Trellis-based parallel stereo matching," Proceedings of IEEE conference on Acoustics, Speech, and Signal Processing, vol. 4, pp.2143-2146, 2000.
  8. X. Sun, X. Mei, S. Jiao, M. Zhou, and H. Wang, "Stereo matching with reliable disparity propagation," Proceedings of IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2011.
  9. L. Zhang, "Fast stereo matching algorithm for intermediate view reconstruction of stereoscopic television images," IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, no. 10, pp. 1259-1270, 2006. https://doi.org/10.1109/TCSVT.2006.882390
  10. L. Do, G. Bravo, S. Zinger, P. H. de With, "GPU-accelerated Real-time Free-viewpoint DIBR for 3DTV," IEEE Transactions on Consumer Electronics, vol. 58, no. 2, May 2012.