Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Wishbowl: Production Case Study of Music Video and Immersive Interactive Concert of Virtual Band Idol Verse'day

Wishbowl: 버추얼 밴드 아이돌 Verse'day 뮤직비디오 및 몰입형 인터랙티브 공연 제작 사례 연구

  • Sebin Lee (The Global School of Media, Soongsil University) ;
  • Gyeongjin Kim (The Global School of Media, Soongsil University) ;
  • Daye Kim (The Global School of Media, Soongsil University) ;
  • Jungjin Lee (The Global School of Media, Soongsil University)
  • 이세빈 (숭실대학교 글로벌미디어학부) ;
  • 김경진 (숭실대학교 글로벌미디어학부) ;
  • 김다예 (숭실대학교 글로벌미디어학부) ;
  • 이정진 (숭실대학교 글로벌미디어학부)
  • Received : 2024.06.15
  • Accepted : 2024.07.05
  • Published : 2024.07.25
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, various virtual avatar music content that showcases singing and dancing have been produced, and as virtual artists gain popularity, offline virtual avatar concerts have also emerged. However, there are few examples of virtual avatar band content where avatars play instruments. In addition, offline virtual avatar concerts using large screens at the front are limited in their ability to utilize the fantastical effects and high degree of freedom unique to virtual reality. In this paper, inspired by these limitations of virtual avatar music content, we introduce the production case of virtual avatar band content and immersive interactive concert of virtual band idol Verse'day. Firstly, we present a case study on creating band performance animations and music videos using motion capture systems and real-time engines. Then, we introduce a production case of an immersive interactive concert using projection mapping technology and a light stick that allows real-time interaction in an offline concert. Finally, based on these production cases, we discussed the future research directions of developing virtual avatar music content creation. We expect that our production cases will inspire the creation of diverse virtual avatar music content and the development of immersive interactive offline virtual avatar concerts in the future.

최근 가상 아바타를 이용하여 가창과 안무를 표현하는 다양한 가상 아바타 음악 콘텐츠가 제작되고 있으며, 가상 아티스트들의 인기가 높아짐에 따라 오프라인에서도 가상 아바타 콘서트를 하는 사례도 등장하고 있다. 하지만 가상 아바타가 악기를 연주하는 밴드 콘텐츠의 제작 사례는 찾아보기 어려우며, 전면의 대형 스크린을 사용하는 오프라인 가상 아바타 콘서트는 가상 현실 특유의 환상적인 연출과 높은 자유도를 활용하기 어렵다는 단점이 있다. 본 논문은 이러한 제한점에 영감을 얻어, 가상 밴드 아이돌 Verse'day의 가상 아바타 밴드 콘텐츠 제작과 몰입형 인터랙티브 공연 제작 사례를 소개한다. 먼저, 본 논문에서는 모션 캡쳐 시스템과 실시간 엔진을 활용해 밴드 연주 애니메이션과 뮤직비디오를 제작한 과정을 소개한다. 이어서, 오프라인 콘서트에서 프로젝션 매핑과 실시간 상호작용 응원봉을 사용해 몰입형 인터랙티브 가상 아바타 공연을 제작한 사례를 소개한다. 마지막으로, 본 논문의 제작 사례를 바탕으로 향후 가상 아바타 음악 콘텐츠 제작을 위한 연구 방향을 제언하였다. 우리는 본 제작 사례가 향후 다양한 가상 아바타 음악 콘텐츠 제작과, 몰입형 인터랙티브 오프라인 가상 아바타 콘서트 제작에 영감을 줄 수 있을 것으로 기대한다.

Keywords

Acknowledgement

본 연구는 과학기술정보통신부 및 정보통신기획평가원의 메타버스 융합대학원 (IITP-2024-RS-2024-00430997, 기여율 20%)과 지역지능화혁신인재양성사업 (IITP-2024-RS-2022-00156360, 기여율 20%)과 문화체육관광부 및 한국콘텐츠 진흥원의 2024 년도 문화체육관광연구개발사업 (연구개발 과제명 : 공연 콘텐츠의 고해상도(8K/16K) 서비스를 위한 AI 기반 영상확장 및 서비스 기술개발, 연구개발과제번호 : RS-2024-00395886, 기여율: 60%)의 지원을 받아 수행되었음. 본 논문의 저자들은 Wishbowl 제작에 도움을 주신 모든 분들께 감사를 표합니다.

References

  1. 김수현 and 유지연, "엔터테인먼트 영역에서의 메타버스 인식과 수용방식 연구 : 에스파(aespa) 팬덤 사례를 중심으로," 한국엔터테인먼트산업학회논문지, vol. 15, no. 7, pp. 1-15, 2021, publisher: 한국엔터테인먼트산업 학회. [Online]. Available: http://www.dbpia.co.kr/journal/articleDetail?nodeId=NODE10622529  10622529
  2. PLAVE 플레이브. Plave. [Online]. Available: https://www.youtube.com/@plave_official 
  3. MAVE: (메이브). Mave. [Online]. Available: https://www.youtube.com/@mave_official 
  4. Z. Lu, C. Shen, J. Li, H. Shen, and D. Wigdor, "More kawaii than a real-person live streamer: Understanding how the otaku community engages with and perceives virtual youtubers," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ser. CHI '21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3411764.3445660 
  5. S. Lee and J. Lee, ""Ju. t'aime" my idol, my streamer: A case study on fandom experience as audiences and creators of vtuber concert," IEEE Access, vol. 11, pp. 31125-31142, 2023.  https://doi.org/10.1109/ACCESS.2023.3252563
  6. S. Lee, D. Kim, and J. Lee, "The effects of viewing formats and song genres on audience experiences in virtual avatar concerts," in Proceedings of the 31st ACM International Conference on Multimedia, ser. MM '23. New York, NY, USA: Association for Computing Machinery, 2023, p. 6978-6988. [Online]. Available: https://doi.org/10.1145/3581783.3612094 
  7. Reliable Business Insights. (2023) Rock instrument market size, cagr, trends 2024-2030. [Online]. Available: https://www.reliablebusinessinsights.com/purchase/866645 
  8. J.-Y. HWANG and M.-Y. KIM. (2024) Silica gel, day6, the rose: Korean music tastes shift to rock. [Online]. Available: https://koreajoongangdaily.joins.com/news/2024-05-13/entertainment/musicPerformance/Silica-Gel-DAY6-The-Rose-Korean-music-tastes-shift-to-rock-/2045266 
  9. OSEN. (2024) Qwer은 어떻게 걸밴드를 대중문화 반열로 올렸나. [Online]. Available: https://biz.chosun.com/entertainment/enter_general/2024/05/24/I3BLUG23WQ67KWIZAF3MWYDKUI/ 
  10. 나혜인. (2024) 플레이브→이세돌, 오프라인 콘서트 삼킨 버추얼돌. [Online]. Available: https://www.sportsq.co.kr/news/articleView.html?idxno=457076 
  11. 고은이. (2024) '버추얼 아이돌' 전성시대... '오프라인 공연'에 팬 10만명 몰렸다. [Online]. Available: https://www.hankyung.com/article/2024041526531 
  12. K. E. Onderdijk, L. Bouckaert, E. Van Dyck, and P.-J. Maes, "Concert experiences in virtual reality environments," Virtual Reality, vol. 27, no. 3, pp. 2383-2396, 2023.  https://doi.org/10.1007/s10055-023-00814-y
  13. Epic Games Inc. (2024) MetaHuman Creator. [Online]. Available: https://metahuman.unrealengine.com 
  14. Justin Bieber. (2021) Wave presents: Justin bieber - an interactive virtual experience. [Online]. Available: https://www.youtube.com/watch?v=UAhGvhvcoyY 
  15. Ariana Grande. (2021) Fortnite presents: Rift tour featuring ariana grande. [Online]. Available: https://www.youtube.com/watch?v=RiM0moNk74o 
  16. S. Lee, G. Lee, S. Han, S. Jeong, and J. Lee, "A simulcast system for live streaming and virtual avatar concerts," Journal of the Korea Computer Graphics Society, vol. 29, no. 2, pp. 21-30, 2023. [Online]. Available: https://doi.org/10.15701/kcgs.2023.29.2.21 
  17. Y. Kang, "Codemiko: an interactive vtuber experience," in SIGGRAPH Asia 2021 Real-Time Live!, ser. SA '21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3478511.3491309 
  18. A. Beacco, R. Oliva, C. Cabreira, J. Gallego, and M. Slater, "Disturbance and Plausibility in a Virtual Rock Concert: A Pilot Study," in 2021 IEEE Virtual Reality and 3D User Interfaces (VR), Mar. 2021, pp. 538-545, iSSN: 2642-5254. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9417644 
  19. T. Kyriakou, M.de la Campa Crespo, A. Panayiotou, Y. Chrysanthou, P. Charalambous, and A. Aristidou, "Virtual instrument performances (vip): A comprehensive review," Computer Graphics Forum, vol. 43, no. 2, p. e15065, 2024. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.15065 
  20. Parable Entertainment Inc. (2023) Isegye festival. [Online]. Available: https://isegye-festival.com 
  21. Statista. (2024) Metaverse Live Entertainment - Worldwid. [Online]. Available: https://www.statista.com/outlook/amo/metaverse/metaverse-live-entertainment/worldwide 
  22. D. Swarbrick, B. Seibt, N. Grinspun, and J. K. Vu- oskoski, "Corona concerts: The effect of virtual con- cert characteristics on social connection and kama muta," Frontiers in Psychology, vol. 12, 2021. [Online]. Available: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2021.648448 
  23. F. Vandenberg, "Put your "hand emotes in the air:" twitch concerts as unsuccessful large-scale interaction rituals," Symbolic Interaction, vol. 45, no. 3, pp. 425-448, 2022. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/symb.605 
  24. C. Cruz-Neira, D. J. Sandin, and T. A. DeFanti, "Surround-screen projection-based virtual reality: the design and implementation of the cave," in Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, ser. SIGGRAPH '93. New York, NY, USA: Association for Computing Machinery, 1993, p. 135-142. [Online]. Available: https://doi.org/10.1145/166117.166134 
  25. 임해중. (2022) 메타버스로 탄생한 버추얼 아이돌 verse'day(벌스데이) 데뷔. [Online]. Available: https://www.news1.kr/articles/?4852942 
  26. Verse'day. (2024) Verse'day(벌스데이) - wishbowl [m/v]. [Online]. Available: https://www.youtube.com/watch?v=h6GKyl106Xs 
  27. Noitom. (2024) Perception neuron studio - professional and precise mocap suit. [Online]. Available: https://neuronmocap.com/pages/perception-neuron-studio-system 
  28. Epic Games Inc. (2024) Unreal engine. [Online]. Available: https://www.unrealengine.com/en-US 
  29. Noitom. (2024) Perception neuron studio gloves. [Online]. Available: https://neuronmocap.com/products/perception-neuron-studio-gloves 
  30. Noitom. (2024) Perception neuron - axis studio motion capture software. [Online]. Available: https://neuronmocap.com/pages/axis-studio 
  31. Noitom. (2024) Perception neuron help center: Perception neuron studio. [Online]. Available: https://support.neuronmocap.com/hc/en-us/articles/4401830416539-Perception-NeuronStudio 
  32. Epic Games Inc. (2024) Live link face for unreal engine. [Online]. Available: https://apps.apple.com/us/app/live-link-face/id1495370836 
  33. Epic Games Inc. (2024) Developing for arkit. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/developing-for-arkit-in-unreal-engine. 
  34. Epic Games Inc. (2024) Ik rig editor. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/ik-rig-in-unreal-engine 
  35. Epic Games Inc. (2024) Ik rig retargeting. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/ik-rig-animation-retargeting-in-unreal-engine 
  36. Epic GamesInc. (2024) Retargeting bipeds with ik rigs. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/animation-blueprint-head-look-at-in-unreal-engine 
  37. Epic Games Inc. (2024) Look at. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unrealengine/animation-blueprint-head-look-at-in-unreal-engine 
  38. Epic Games Inc. (2024) Creating visual effects [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/creating-visual-effects-in-niagara-for-unreal-engine 
  39. Epic Games Inc. (2024) Cine camera actor. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/cinematic-cameras-in-unreal-engine 
  40. Epic GamesInc. (2024) Sequencer overview. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/unreal-engine-sequencer-movie-tool-overview 
  41. Epic Games Inc. (2024) Camera rigs. [Online]. Available: https://dev.epicgames.com/documentation/en-us/unreal-engine/camera-jibs-and-dollies-in-unreal-engine 
  42. Verse'day. (2024) Verse'day - Wishbowl [M/V]. [Online]. Available: https://www.youtube.com/watch?v=h6GKyl106Xs 
  43. H. Yakura and M. Goto, "Enhancing participation experience in vr live concerts by improving motions of virtual audience avatars," in 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020, pp. 555-565. 
  44. W. Oshiro, M. Kohjima, H. Kataoka, M. Yokoyama, Y. Nambu, M. Makiguchi, R. Mochizuki, and R. Yamamoto, "Evaluation of virtual audience synchronized with cheering motion of the user's light stick during music concerts," in HCI International 2024 Posters. Cham: Springer Nature Switzerland, 2024, pp. 332-340. 
  45. Epic Games Inc. (2024) Rendering to multiple displays with ndisplay. [Online]. Available: https://dev.epicgames.com/documentation/ko-kr/unreal-engine/rendering-to-multiple-displays-with-ndisplay-in-unreal-engine 
  46. J. Lee, S. Lee, Y. Kim, and J. Noh, "Screenx: Public immersive theatres with uniform movie viewing experiences," IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 2, pp. 1124-1138, 2017. https://doi.org/10.1109/TVCG.2016.2532327