Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Panorama Image Stitching Using Sythetic Fisheye Image

Synthetic fisheye 이미지를 이용한 360° 파노라마 이미지 스티칭

  • Kweon, Hyeok-Joon (Department of Electronics Engineering, Chungnam National University) ;
  • Cho, Donghyeon (Department of Electronics Engineering, Chungnam National University)
  • Received : 2021.11.23
  • Accepted : 2021.12.27
  • Published : 2022.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, as VR (Virtual Reality) technology has been in the spotlight, 360° panoramic images that can view lively VR contents are attracting a lot of attention. Image stitching technology is a major technology for producing 360° panorama images, and many studies are being actively conducted. Typical stitching algorithms are based on feature point-based image stitching. However, conventional feature point-based image stitching methods have a problem that stitching results are intensely affected by feature points. To solve this problem, deep learning-based image stitching technologies have recently been studied, but there are still many problems when there are few overlapping areas between images or large parallax. In addition, there is a limit to complete supervised learning because labeled ground-truth panorama images cannot be obtained in a real environment. Therefore, we produced three fisheye images with different camera centers and corresponding ground truth image through carla simulator that is widely used in the autonomous driving field. We propose image stitching model that creates a 360° panorama image with the produced fisheye image. The final experimental results are virtual datasets configured similar to the actual environment, verifying stitching results that are strong against various environments and large parallax.

최근 VR (Virtual Reality) 기술이 주목받기 시작하면서 생동감 넘치는 VR 컨텐츠를 볼 수 있는 360° 파노라마 영상이 많은 관심을 받고 있다. 이미지 스티칭 기술은 360° 파노라마 영상을 제작하는데 주요한 기술로서 많은 연구가 활발하게 이루어지고 있다. 일반적인 스티칭 알고리즘은 특징점 기반 이미지 스티칭을 기반으로 한다. 하지만 기존의 특징점 기반 이미지 스티칭 방법들은 특징점에 크게 영향을 받는다는 문제가 존재한다. 이러한 문제를 해결하기 위해서 최근에는 딥러닝 기반의 이미지 스티칭 기술들이 연구되고 있지만 이미지 간의 겹치는 영역이 거의 없거나 큰 시차가 존재할 때 여전히 많은 문제점이 존재한다. 또한 실제 환경에서는 라벨링 된 정답 파노라마 영상을 얻을 수 없으므로 완전한 지도학습에 한계가 존재한다. 따라서 자율주행분야에 많이 이용되는 칼라(Carla) 시뮬레이터를 통해 카메라 센터가 다른 3개의 fisheye 이미지와 그에 대응되는 정답 영상을 제작하였다. 우리는 제작한 fisheye 영상으로360° 파노라마 영상을 만드는 이미지 스티칭 모델을 제안한다. 최종 실험 결과로는 실제 환경과 비슷하게 구성한 가상의 데이터 세트로 다양한 환경과 큰 시차에도 강인한 스티칭 결과를 검증한다.

Keywords

Acknowledgement

This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2018-0-00207, Immersive Media Research Laboratory) and the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) under Grant 2020R1F1A1065573.

References

  1. Harris, C., & Stephens, M. A combined corner and edge detector. In Alvey vision conference, Vol. 15, No. 50, pp. 10-5244, August 1988.
  2. Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. ORB: An efficient alternative to SIFT or SURF. In International conference on computer vision, pp. 2564-2571 , November 2011.
  3. Bay, H., Tuytelaars, T., & Van Gool, L. Surf: Speeded up robust features. In European conference on computer vision, pp. 404-417, May 2006.
  4. Rosten, E., & Drummond, T. Machine learning for high-speed corner detection. In European conference on computer vision, pp. 430-443, May 2006.
  5. Lowe, D. G. Distinctive image features from scale-invariant keypoints. In International journal of computer vision, 60(2), 91-110, 2004. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  6. Fischler, M. A., & Bolles, R. C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM, 24(6), 381-395, 1981. https://doi.org/10.1145/358669.358692
  7. Burt, Peter J., and Edward H. Adelson. "A multiresolution spline with application to image mosaics." In ACM Transactions on Graphics (TOG) 2.4, 217-236, 1983 . https://doi.org/10.1145/245.247
  8. Perez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. In ACM SIGGRAPH, pp. 313-318, 2003 .
  9. Gao, J., Kim, S. J., & Brown, M. S. Constructing image panoramas using dual-homography warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 49-56, June 2011.
  10. Lin, W. Y., Liu, S., Matsushita, Y., Ng, T. T., & Cheong, L. F. Smoothly varying affine stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 345-352, June 2011
  11. Zaragoza, J., Chin, T. J., Brown, M. S., & Suter, D. As-projective-as-possible image stitching with moving DLT. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2339-2346, 2013.
  12. Chang, C. H., Sato, Y., & Chuang, Y. Y. Shape-preserving half-projective warps for image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3254-3261, 2014.
  13. Lee, K. Y., & Sim, J. Y. Warping residual based image stitching for large parallax. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8198-8206, 2020.
  14. Liu, F., Gleicher, M., Jin, H., & Agarwala, A. Content-preserving warps for 3D video stabilization. In ACM Transactions on Graphics (ToG), 28(3), 1-9, 2009.
  15. Zhang, F., & Liu, F. Parallax-tolerant image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262-3269, 2014.
  16. Lin, K., Jiang, N., Cheong, L. F., Do, M., & Lu, J. Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In European conference on computer vision, pp. 370-385, October 2016.
  17. Hejazifar, H., & Khotanlou, H. Fast and robust seam estimation to seamless image stitching. In Signal, Image and Video Processing, 12(5), 885-893, 2018. https://doi.org/10.1007/s11760-017-1231-3
  18. Hoang, V. D., Tran, D. P., Nhu, N. G., & Pham, V. H. Deep feature extraction for panoramic image stitching. In Asian Conference on Intelligent Information and Database Systems, pp. 141-151, March 2020.
  19. Shi, Z., Li, H., Cao, Q., Ren, H., & Fan, B. An image mosaic method based on convolutional neural network semantic features extraction. In Journal of Signal Processing Systems, 92(4), 435-444, 2020. https://doi.org/10.1007/s11265-019-01477-2
  20. Wang, L., Yu, W., & Li, B. (2020, June). Multi-scenes image stitching based on autonomous driving. In IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Vol. 1, pp. 694-698, June 2020.
  21. Li, J., Zhao, Y., Ye, W., Yu, K., & Ge, S. Attentive Deep Stitching and Quality Assessment for 360° Omnidirectional Images. In IEEE Journal of Selected Topics in Signal Processing, 14(1), 209-221, 2019. https://doi.org/10.1109/jstsp.2019.2953950
  22. Song, D. Y., Um, G. M., Lee, H. K., & Cho, D. End-to-End Image Stitching Network via Multi-Homography Estimation. In IEEE Signal Processing Letters, 28, 763-767, 2021. https://doi.org/10.1109/LSP.2021.3070525
  23. Nie, L., Lin, C., Liao, K., Liu, M., & Zhao, Y. A view-free image stitching network based on global homography. In Journal of Visual Communication and Image Representation, 73, 102950, 2020. https://doi.org/10.1016/j.jvcir.2020.102950
  24. Sekkat, A. R., Dupuis, Y., Vasseur, P., & Honeine, P. The omniscape dataset. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1603-1608, May 2020.
  25. Berenguel-Baeta, B., Bermudez-Cameo, J., & Guerrero, J. J. OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision. In Sensors, 20(7), 2066, 2020. https://doi.org/10.3390/s20072066
  26. Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006, October). A toolbox for easily calibrating omnidirectional cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5695-5701, October 2006.
  27. Mittal, Anish, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. In IEEE Signal Processing Letters (SPL), pp. 209-212. 20.3 (2012) https://doi.org/10.1109/LSP.2012.2227726
  28. Zhu, Hancheng, et al. MetaIQA: Deep meta-learning for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.