Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Exploratory Experimental Analysis for 2D to 3D Generation

2D to 3D 창의적 생성을 위한 탐색적 실험 분석

  • Hyeongrae Cho (Department of Electronic IT Media Engineering, Seoul National University of Science and Technology) ;
  • Ilsik Chang (Department of Electronic IT Media Engineering, Seoul National University of Science and Technology) ;
  • Hyunseok Kang (Department of Electronic IT Media Engineering, Seoul National University of Science and Technology) ;
  • Youngchan Go (Department of Electronic IT Media Engineering, Seoul National University of Science and Technology) ;
  • Gooman Park (Department of Electronic IT Media Engineering, Seoul National University of Science and Technology)
  • 조형래 (서울과학기술대학교 전자IT미디어공학과) ;
  • 장일식 (서울과학기술대학교 전자IT미디어공학과) ;
  • 강현석 (서울과학기술대학교 전자IT미디어공학과) ;
  • 고영찬 (서울과학기술대학교 전자IT미디어공학과) ;
  • 박구만 (서울과학기술대학교 전자IT미디어공학과)
  • Received : 2022.11.29
  • Accepted : 2023.01.24
  • Published : 2023.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

Deep learning has made rapid progress in recent years and is affecting various fields and industries. The art field cannot be an exception, and in this paper, we would like to explore and experiment and analyze research fields that creatively generate 2D images in 3D from a visual arts and engineering perspective. To this end, the original image of the domestic artist is learned through GAN or Diffusion Models, and then converted into 3D using 3D conversion software and deep learning. And we compare the results with prior algorithms. After that, we will analyze the problems and improvements of 2D to 3D creative generation.

딥러닝은 최근 몇 년 동안 비약적인 발전을 하였고 다양한 분야 및 산업에 영향을 주고 있다. 예술영역도 예외일 수는 없는데 본 논문에서는 시각예술·공학적 관점에서 2D 이미지를 3D로 창의적으로 생성하는 방법을 실험하고자 한다. 이를 위해 국내 아티스트 원본 이미지를 GAN 또는 Diffusion Models로 학습시킨 후 3D 변환 소프트웨어와 딥러닝을 활용하여 3D로 변환하고 그 결과를 선행연구 알고리즘과 비교 실험함으로써 2D to 3D 창의적 생성의 문제점과 개선점을 분석하고자 한다.

Keywords

Acknowledgement

이 글은 2022년도 과학기술정보통신부의 재원으로 정보통신기획평가원의 지원을 받아 수행된 연구임 (No.2021-0-00751, 0.5mm급 이하 초정밀 가시·비가시 정보 표출을 위한 다차원 시각화 디지털 트윈 프레임워크 기술개발).

References

  1. Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka,Wadim Kehl and Adrien Gaidon,Differentiable Rendering, 2020, https://arxiv.org/pdf/2006.12057.pdf
  2. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, 2020, doi: https://doi.org/10.1145/3503250
  3. Artist's statement, https://en.wikipedia.org/wiki/Artist%27s_statement (Accessed November. 17, 2022)
  4. Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt, STYLENERF: A STYLE-BASED 3D-AWARE GENERATOR FOR HIGH-RESOLUTION IMAGE SYNTHESIS, 2021, https://arxiv.org/pdf/2110.08985.pdf
  5. Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole, Zero-Shot Text-Guided Object Generation with Dream Fields, 2022. doi: https://doi.org/10.1109/cvpr52688.2022.00094
  6. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, Learning Transferable Visual Models From Natural Language Supervision, 2021, https://arxiv.org/pdf/2103.00020.pdf
  7. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel, IMAGENET-TRAINED CNNS ARE BIASED TOWARDS TEXTURE; INCREASING SHAPE BIAS IMPROVES ACCURACY AND ROBUSTNESS, 2019, https://openreview.net/pdf?id=Bygh9j09KX
  8. Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Hao Su, NeuTex: Neural Texture Mapping for Volumetric Neural Rendering, 2021, https://arxiv.org/pdf/2103.00762.pdf
  9. Lukas Hollein, Justin Johnson, Matthias Niessner, Technical University of Munich, University of Michigan, StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions, 2022, doi: https://doi.org/10.1109/cvpr52688.2022.00610
  10. Aysegul Dundar, Jun Gao, Andrew Tao, Bryan Catanzaro, Fine Detailed Texture Learning for 3D Meshes with Generative Models, 2022, https://arxiv.org/pdf/2203.09362.pdf
  11. Zhiqin Chen, Vladimir G. Kim, Matthew Fisher, Noam Aigerman, Hao Zhang, Siddhartha Chaudhuri, Simon Fraser University, Adobe Research, IIT Bombay, DECOR-GAN: 3D Shape Detailization by Conditional Refinement, 2021. doi: https://doi.org/10.1109/cvpr46437.2021.01548
  12. Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, Olga Sorkine-Hornung, ETH Zurich, Adobe Research, IIT Bombay, Neural Cages for Detail-Preserving 3D Deformations, 2020. doi: https://doi.org/10.1109/cvpr42600.2020.00015
  13. Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein, Efficient Geometry-aware 3D Generative Adversarial Networks, 2022. doi: https://doi.org/10.1109/cvpr52688.2022.01565
  14. Xudong Xu, Xingang Pan, Dahua Lin, Bo Dai, Generative Occupancy Fields for 3D Surface-Aware Image Synthesis, "Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI), Nov 2021. doi: https://doi.org/10.48550/arXiv.2111.00969
  15. Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger, Max Planck Institute for Intelligent Systems, Tubingen, University of Tubingen, Amazon, Tubingen, ETAS GmbH, Bosch Group, Stuttgart, Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision, "Computer Vision and Pattern Recognition (cs.CV), jun 2020. doi: https://doi.org/10.1109/cvpr42600.2020.00356
  16. Zhiqin Chen, Kangxue Yin, Sanja Fidler, AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis, 2022. doi: https://doi.org/10.1109/cvpr52688.2022.00152
  17. THOMAS MuLLER, ALEX EVANS, CHRISTOPH SCHIED, ALEXANDER KELLER, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, 2022. doi: https://doi.org/10.1145/3528223.3530127
  18. Zhiqin Chen, Thomas Funkhouser, Peter Hedman, Andrea Tagliasacchi, MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures, 2022, https://arxiv.org/pdf/2208.00277.pdf
  19. Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Muller, Sanja Fidler, Extracting Triangular 3D Models, Materials, and Lighting From Images, 2022, doi: https://doi.org/10.1109/cvpr52688.2022.00810
  20. Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall, DREAMFUSION: TEXT-TO-3D USING 2D DIFFUSION, 2022, https://arxiv.org/pdf/2209.14988.pdf
  21. Gwanghyun Kim, Taesung Kwon, Jong Chul Ye,DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation, "CVPR, Jun 2022. doi; https://doi.org/10.1109/cvpr52688.2022.00246
  22. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Image Style Transfer Using Convolutional Neural Networks,"CVPR, Jun 2016. doi: https://doi.org/10.1109/cvpr.2016.265
  23. RUIHUI LI, XIANZHI LI, KA-HEI HUI, and CHI-WING FU, SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation, "ACM Transactions on Graphics, Vol.40, No.4, pp.1-12, Aug 2021. doi: https://doi.org/10.5909/JBE.2020.25.5.776
  24. RANA HANOCKA, GAL METZER, RAJA GIRYES, DANIEL COHEN-OR, Point2Mesh: A Self-Prior for Deformable Meshes, "ACM Transactions on Graphics,Vol.39, No.4, Aug 2020. doi: https://doi.org/10.1145/3386569.3392415
  25. Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao, CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields, "CVPR, Mar 2022. doi: https://doi.org/10.48550/arXiv.2112.05139
  26. Diffusion model, https://en.wikipedia.org/wiki/Diffusion_model (Accessed November. 17, 2022)
  27. Narek Tumanyan, Michal Geyer, Shai Bagon, Tali Dekel, Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation,Tue, 22 Nov 2022. doi: https://doi.org/10.48550/arXiv.2211.12572
  28. Jonathan Ho, Ajay Jain, Pieter Abbeel, Denoising Diffusion Probabilistic Models, Wed, 16 Dec 2020. doi: https://doi.org/10.48550/arXiv.2006.11239
  29. Jiaming Song, Chenlin Meng, Stefano Ermon, Denoising Diffusion Implicit Models, Wed, 5 Oct 2022. doi: https://doi.org/10.48550/arXiv.2010.02502
  30. Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, Andreas Geiger, Shape As Points: A Differentiable Poisson Solver, "NeurIPS, Mon 7 Jun 2021. doi: https://doi.org/10.48550/arXiv.2106.03452
  31. Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li, Shanghai AI Laboratory, Peking University, The Chinese University of Hong Kong, PointCLIP: Point Cloud Understanding by CLIP, "CVPR, Jun 2021. doi: https://doi.org/10.1109/cvpr52688.2022.00836