The Journal of Korea Robotics Society (로봇학회논문지)

Korea Robotics Society (한국로봇학회)
권호 표지
DOI QR코드

DOI QR Code

Distance and Entropy Based Image Viewpoint Selection for Accurate 3D Reconstruction with NeRF

NeRF의 정확한 3차원 복원을 위한 거리-엔트로피 기반 영상 시점 선택 기술

  • Jinwon Choi (Computer Science and Engineering Department, Seoul National University of Science and Technology (SEOULTECH)) ;
  • Chanho Seo (Computer Science and Engineering Department, Seoul National University of Science and Technology (SEOULTECH)) ;
  • Junhyeok Choi (Computer Science and Engineering Department, Seoul National University of Science and Technology (SEOULTECH)) ;
  • Sunglok Choi (Computer Science and Engineering Department, Seoul National University of Science and Technology (SEOULTECH))
  • Received : 2023.10.31
  • Accepted : 2023.12.18
  • Published : 2024.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a new approach with a distance-based regularization to the entropy applied to the NBV (Next-Best-View) selection with NeRF (Neural Radiance Fields). 3D reconstruction requires images from various viewpoints, and selecting where to capture these images is a highly complex problem. In a recent work, image acquisition was derived using NeRF's ray-based uncertainty. While this work was effective for evaluating candidate viewpoints at fixed distances from a camera to an object, it is limited when dealing with a range of candidate viewpoints at various distances, because it tends to favor selecting viewpoints at closer distances. Acquiring images from nearby viewpoints is beneficial for capturing surface details. However, with the limited number of images, its image selection is less overlapped and less frequently observed, so its reconstructed result is sensitive to noise and contains undesired artifacts. We propose a method that incorporates distance-based regularization into entropy, allowing us to acquire images at distances conducive to capturing both surface details without undesired noise and artifacts. Our experiments with synthetic images demonstrated that NeRF models with the proposed distance and entropy-based criteria achieved around 50 percent fewer reconstruction errors than the recent work.

Keywords

Acknowledgement

This research was supported by MSIT/NRF Grant for Bridge Convergence R&D Program (AI-based Localization and Path Planning on 3D Building Surfaces; 2021M3C1C3096810) and CHA/NRICH Grant for a R&D Program (Development of Ultra-High Resolution Gigapixel 3D Data Generation Technology; 2021A02P02-001)

References

  1. S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza, "An Information Gain Formulation for Active Volumetric 3D Reconstruction," 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, pp. 3477-3484, May., 2016, DOI: 10.1109/ICRA.2016.7487527. 
  2. L. hou, X. Chen, K. Lan, R. RasMmussen, and J. Roberts, "Volumetric Next Best View by 3D Occupancy Mapping Using Markov Chain Gibbs Sampler for Precise Manufacturing," IEEE Access, vol. 7, pp. 121949-121960, Aug., 2019, DOI: 10.1109/ACCESS.2019.2935547. 
  3. D. Peralta, J. Casimiro, A. M. Nilles, J. A. Aguilar, R. Atienza, and R. Cajote, "Next-Best View Policy for 3D Reconstruction," arXiv:2008.12664, pp 558-573, Jan., 2020, DOI: 10.48550/arXiv.2008.12664. 
  4. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," European Conference on Computer Vision, vol. 123456, pp. 405-421, Nov., 2020, DOI: 10.1007/978-3-030-58452-8_24. 
  5. S. Lee, L. Chen, J. Wang, A. Liniger, S. Kumar, and F. Yu, "Uncertainty Guided Policy for Active Robotic 3D Reconstruction Using Neural Radiance Fields," IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12070-12077, Oct., 2022, DOI: 10.1109/LRA.2022.3212668. 
  6. L. Jin, X. Chen, J. Ruckin, and M. Popovic, "NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering," arXiv:2303.01284, 2023, DOI: 10.48550/arXiv.2303.01284. 
  7. Y. Ran , J. Zeng , S. He, J. Chen , L. Li , Y. Chen , G. Lee, and Q. Ye, "NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction With Implicit Neural Representations," IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 1125-1132, Feb., 2023, DOI: 10.1109/LRA.2023.3235686. 
  8. L. M. Wong, C. Dumont, and M. A. Abidi, "Next-best-view algorithm for object reconstruction," Sensor Fusion and Decentralized Control in Robotic Systems, vol. 3523, pp. 191-200, Oct., 1998, DOI: 10.1117/12.327001. 
  9. C. E. Shannon, "A mathematical theory of communication," The Bell System Technical Journal, vol. 27, no. 3, pp. 379-423, Jul., 1948, DOI: 10.1002/j.1538-7305.1948.tb01338.x. 
  10. J. Tang, H. Zhou, X. Chen, T. Hu, E. Ding, J. Wang, and G. Zeng, "Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement," arXiv:2303.02091, 2023, DOI: 10.48550/arXiv.2303.02091. 
  11. N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixel2mesh: Generating 3d mesh models from single rgb images," European conference on computer vision, vol. 11215, pp. 52-67, Oct., 2018, DOI: 10.1007/978-3-030-01252-6_4. 
  12. T. Muller, A. Evans, C. Schied, and A. Keller, "Instant Neural Graphics Primitives with a Multiresolution Hash Encoding," ACM Transactions on Graphics, vol. 41, no. 4, pp. 1-15, Jul., 2022, DOI: 10.1145/3528223.3530127.