KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Real-time 3D Audio Downmixing System based on Sound Rendering for the Immersive Sound of Mobile Virtual Reality Applications

  • Hong, Dukki (Department of Computer Engineering, Sejong University) ;
  • Kwon, Hyuck-Joo (School of Electrical & Electronic Engineering, Yonsei University) ;
  • Kim, Cheong Ghil (Department of Computer Science, Namseoul University) ;
  • Park, Woo-Chan (Department of Computer Engineering, Sejong University)
  • Received : 2017.09.27
  • Accepted : 2018.05.05
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Eight out of the top ten the largest technology companies in the world are involved in some way with the coming mobile VR revolution since Facebook acquired Oculus. This trend has allowed the technology related with mobile VR to achieve remarkable growth in both academic and industry. Therefore, the importance of reproducing the acoustic expression for users to experience more realistic is increasing because auditory cues can enhance the perception of the complicated surrounding environment without the visual system in VR. This paper presents a audio downmixing system for auralization based on hardware, a stage of sound rendering pipelines that can reproduce realiy-like sound but requires high computation costs. The proposed system is verified through an FPGA platform with the special focus on hardware architectural designs for low power and real-time. The results show that the proposed system on an FPGA can downmix maximum 5 sources in real-time rate (52 FPS), with 382 mW low power consumptions. Furthermore, the generated 3D sound with the proposed system was verified with satisfactory results of sound quality via the user evaluation.

Keywords

References

  1. K. S. Hale and K. M. Stanney (ed.), "Handbook of virtual environments: Design, implementation, and applications," CRC Press: Boca Raton, US, 2014.
  2. M. Vorlander, "Auralization: Fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual reality," Springer: Berlin, Germany, 2008.
  3. A. Febrettia, A. Nishimotoa, T. Thigpena, J. Talandisa, L. Longa, J. Pirtlea, T. Peterkaa, A. Verloa, M. Browna, D. Plepysa, D. Sandina, L. Renambota, A. Johnsona, and J. Leigha, "Cave2: a hybrid reality environment for immersive simulation and information analysis," in Proc. of Proceedings of SPIE Electronic Imaging 2013, Burlingame, CA, USA, 3-7 Feb. 2013.
  4. L. Antani, N. Galoppo, A. Lake, and A. Peleg, "Next-gen sound rendering with OpenCL," in Proc. of Proceedings of ACM SIGGRAPH 2011 BOF OpenCL, Vancouver, BC, Canada, 7-11 Aug. 2011.
  5. B. Hook and T. Smurdon, "An intro to virtual reality audio," in Proc. of Game Developers Conference (GDC) 2015, San Francisco, CA, USA, 2-6 Mar. 2015.
  6. N. Ward-Foxton, "Environmental audio and processing for VR," in Proc. of Game Developers Conference (GDC) 2015, San Francisco, CA, USA, 2-6 Mar. 2015.
  7. A. Chandak, "Effcient geometric sound propagation using visibility culling," Ph.D. dissertation, Department of Computer Science, University of North Carolina at Chapel Hill, NC, USA, 2011.
  8. S. Borkar and A. A. Chien, "The future of microprocessors," Communications of the ACM, vol. 54, no. 5, pp. 67-77, May 2011. https://doi.org/10.1145/1941487.1941507
  9. Dolby, "Dolby ATMOS,"
  10. B. V. Daele and W. V. Baelen, "Professional workflow white paper," Auro Technologies NV, 2011.
  11. AM3D, "ZIRENE 3D: Positional audio,"
  12. Khronos, "OpenSL ES 1.1 specification,"
  13. Oculus, "Oculus audio SDK guide,"
  14. M. T. Taylor, A. Chandak, L. Antani, and D. Manocha, "RESound: Interactive sound rendering for dynamic virtual environments," in Proc. of Proceedings of the 17th ACM international conference on Multimedia (MM 2009), Beijing, China, 19-24 Oct. 2009.
  15. N. Raghuvanshi, J. Snyder, R. Mehra, M. Lin, and N. Govindaraju, "Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes," ACM Transactions on Graphics, vol. 29, no. 4, pp. 68:1-11, Jul. 2010.
  16. R. Mehra, N. Raghuvanshi, L. Antani, A. Chandak, S. Curtis, and D. Manocha, "Wave-based sound propagation in large open scenes using an equivalent source formulation," ACM Transactions on Graphics, vol. 32, no. 2, pp. 19:1-13, Apr. 2013.
  17. R. Mehra, A. Rungta, A. Golas, M. Lin, and D. Manocha, "WAVE: Interactive wave-based sound propagation for virtual environments," IEEE Transactions on Visualization and Computer Graphics, vol. 21, no. 4, pp. 434-442, Jan. 2015. https://doi.org/10.1109/TVCG.2015.2391858
  18. M. Vorlander, "Simulation of the transient and steady-state sound propagation in rooms using a new combined ray-tracing/image-source algorithm," The Journal of the Acoustical Society of America, vol. 86, no. pp. 172-178, 1989. https://doi.org/10.1121/1.398336
  19. C. Schissler and D. Manocha, "Interactive sound propagation and rendering for large multi-source scenes," ACM Transactions on Graphics, vol. 36, no. 1, pp. 2:1-12, Feb. 2017.
  20. C. Cao, Z. Ren, C. Schissler, D. Manocha, and K. Zhou, "Interactive sound propagation with bidirectional path tracing," ACM Transactions on Graphics, vol. 35, no. 6, pp. 180:1-11, Nov. 2016.
  21. Nvidia, "NVIDIA VRWorks audio,"
  22. AMD, "AMD TrueAudio Next (TAN),"
  23. Valve, "Steam Audio,"
  24. C. Schissler and D. Manocha, "GSound: Interactive sound propagation for games," in Proc. of Proceedings of Audio Engineering Society 41st International Conference: Audio for Games, London, UK, 2-4 Feb. 2011.
  25. S. Butterworth, "On the theory of filter amplifiers," Experimental Wireless and the Wireless Engineer, vol. 7, no. 6, pp.536-541, Oct. 1930.
  26. B. Luo, W. Shichuang, and Z. Nanrun, "Flexible design method for multi-repeater wireless power transfer system based on coupled resonator bandpass filter model," IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 61, no. 11, pp. 3288-3297, Nov. 2014. https://doi.org/10.1109/TCSI.2014.2327331
  27. G. Garcia, "Optimal filter partition for efficient convolution with short input/output delay," in Proc. of 113th Convention of the Audio Engineering Society, Scotts Valley, CA, 5-8 Oct. 2002.
  28. M. R. Schroeder, "Natural sounding artificial reverberation," Journal of the Audio Engineering Society, vol. 10, no. 3, pp.219-223, Jul. 1962.
  29. Xilinx, "Xilinx 7 Series FPGAs Data Sheet: Overview,"
  30. Xilinx, "Xilinx Power Estimator: 7 Series and Zynq-7000 Ver. 2017.02,"
  31. CyberLink, "CyberLink PowerDVD17 User Guide,"
  32. D. Hong, T.-H. Lee, Y. Joo, and W.-C. Park, "Real-time sound propagation hardware accelerator for immersive virtual reality 3D audio," in Proc. of Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D 2017), San Francisco, CA, USA, 25-27 Feb. 2017.
  33. The Wall Street Journal, "Augmented and Virtual Reality: A New Vision,"

Cited by

  1. 컨조인트 분석을 통한 집단별 VR게임콘텐츠의 개발 전략에 관한 연구 vol.21, pp.1, 2018, https://doi.org/10.7472/jksii.2020.21.1.137