Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Gadget Arms: Interactive Data Visualization using Hand Gesture in Extended Reality

가젯암: 확장현실을 위한 손 제스처 기반 대화형 데이터 시각화 시스템

  • Choi, JunYoung (Ulsan National Institute of Science and Technology (UNIST)) ;
  • Jeong, HaeJin (Ulsan National Institute of Science and Technology (UNIST)) ;
  • Jeong, Won-Ki (Ulsan National Institute of Science and Technology (UNIST))
  • Received : 2019.05.21
  • Accepted : 2019.05.31
  • Published : 2019.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

Extended Reality (XR), such as virtual and augmented reality, has huge potential for immersive data visualization and analysis. In XR, users can interact with data and other users realistically by navigating the shared virtual space, allowing for more intuitive data analysis. However, creating a visualization in XR also poses a challenge because complicated, low-level programming is required, which hinders broad adaptation in visual analytics. This paper proposes an interactive visualization authoring tool based on hand gesture for immersive data visualization-Gadget Arms. The proposed system provides a novel user interaction to create and place visualization in the 3D virtual world. This simple, but intuitive, user interaction enables user designs the entire visualization space in the XR without using a host computer and low-level programming. Our user study also confirmed that the proposed user interaction significantly improves the usability of the visualization authoring tool.

가상 및 증강현실과 같은 확장현실(XR: Extended Reality)은 몰입형 데이터 시각화 및 분석을 위한 거대한 잠재력을 가지고 있다. 확장현실에서 사용자는 3차원 가상 공간을 활용하여 데이터 및 여러 사용자와 사실적으로 상호 작용할 수 있으므로 보다 직관적인 데이터 분석이 가능하다. 그러나 확장현실의 장점을 제대로 활용하기 위한 시각화를 생성하는 것은 복잡한 프로그래밍이 필요하기 때문에 상당히 어려운 일이다. 본 논문에서는 손 제스처만으로 XR을 위한 시각화 전 과정을 수행하고, 모든 3차원 공간을 활용할 수 있는 가젯암(Gadget Arms) 시스템을 제안한다. 이 시스템에서 분석가는 데스크탑 기반 환경이나 복잡한 프로그래밍을 사용할 필요 없이 XR 기기를 착용한 상태에서 직관적으로 전체 시각화 공간을 디자인할 수 있으며, 가상의 손을 임의의 3차원 공간에 자유자재로 위치시킬 수 있는 인터랙션을 통해 거대한 3차원 공간을 효과적으로 활용할 수 있다. 우리는 사용자 연구를 통해 제안된 시스템이 시각화 제작과 공간 활용의 사용성을 크게 향상시시키는 것을 확인하였다.

Keywords

References

  1. Inspector Gadget. https://en.wikipedia.org/wiki/Inspector_Gadget.
  2. Leap Motion. https://www.leapmotion.com/.
  3. Microsoft Docs. https://docs.microsoft.com/en-us/windows/mixed-reality/gestures
  4. Unity. https://unity3d.com/
  5. Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes. International Journal of Computer Vision 126, 9, 961-972, 2018. https://doi.org/10.1007/s11263-018-1070-x
  6. Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 185, 2018.
  7. Benjamin Bach, Ronell Sicat, Hanspeter Pfister, and Aaron Quigley. Drawing into the AR-CANVAS: Designing embedded visualizations for augmented reality. In Workshop on Immersive Analytics, IEEE Vis, 2017.
  8. Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. Normaltouch and texturetouch: High-fidelity 3d haptic shape rendering on handheld virtual reality controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 717-728, 2016.
  9. Simon Butscher, Sebastian Hubenschmid, Jens Muller, Johannes Fuchs, and Harald Reiterer. Clusters, Trends, and Outliers: How Immersive Technologies Can Facilitate the Collaborative Analysis of Multidimensional Data. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 90, 12 pages. 2018.
  10. Yeonjoo Cha and Rohae Myung. Extended Fitts' law for 3D pointing tasks using 3D target arrangements. International Journal of Industrial Ergonomics 43, 4, 350-355, 2013. https://doi.org/10.1016/j.ergon.2013.05.005
  11. Inrak Choi, Eyal Ofek, Hrvoje Benko, Mike Sinclair, and Christian Holz. CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 654. 2018.
  12. Maxime Cordeil, Andrew Cunningham, Tim Dwyer, Bruce H. Thomas, and Kim Marriott. 2017. ImAxes: Immersive Axes As Embodied Affordances for Interactive Multivariate Data Visualisation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 71-83. https://doi.org/10.1145/3126594.3126613
  13. Maxime Cordeil, Tim Dwyer, Karsten Klein, Bireswar Laha, Kim Marriott, and Bruce H Thomas. Immersive collaborative analysis of network connectivity: Cave-style or head-mounted display? IEEE transactions on visualization and computer graphics 23, 1 (2017), 441-450. 2017. https://doi.org/10.1109/TVCG.2016.2599107
  14. Ciro Donalek, S. George Djorgovski, Scott Davidoff, Alex Cioc, Anwell Wang, Giuseppe Longo, Jeffrey S. Norris, Jerry Zhang, Elizabeth Lawler, Stacy Yeh, Ashish Mahabal, Matthew J. Graham, and Andrew J. Drake. Immersive and collaborative data visualization using virtual reality platforms. 2014 IEEE International Conference on Big Data (Big Data), 609-614. 2014.
  15. Adrien Fonnet, Florian Melki, Yannick Prie, Fabien Picarougne, and Gregoire Cliquet. Immersive Data Exploration and Analysis. In Student Interaction Design Research conference. 2018.
  16. Joon Hyub Lee, Sang-Gyun An, Yongkwan Kim, and Seok-Hyung Bae. Projective Windows: Bringing Windows in Space to the Fingertip. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 218. 2018.
  17. James R Lewis. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of HumanComputer Interaction 7, 1, 57-78. 1995. https://doi.org/10.1080/10447319509526110
  18. Martin Luboschik, Philip Berger, and Oliver Staadt. On spatial perception issuesin augmented reality based immersive analytics. In Proceedings of the 2016 ACM Companion on Interactive Surfaces and Spaces. ACM, 47-53. 2016.
  19. Longfei Ma, Zhe Zhao, Fang Chen, Boyu Zhang, Ligong Fu, and Hongen Liao. Augmented reality surgical navigation with ultrasound-assisted registration for pedicle screw placement: a pilot study. International journal of computer assisted radiology and surgery 12, 12, 2205-2215. 2017. https://doi.org/10.1007/s11548-017-1652-z
  20. Patrick Millais, Simon L. Jones, and Ryan Kelly. Exploring Data in Virtual Reality: Comparisons with 2D Data Visualizations. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW007, 6 pages. 2018.
  21. Patrick Millais, Simon L Jones, and Ryan Kelly. Exploring Data in Virtual Reality: Comparisons with 2D Data Visualizations. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW007. 2018.
  22. Ekaterina Olshannikova, Aleksandr Ometov, Yevgeni Koucheryavy, and Thomas Olsson. Visualizing Big Data with augmented and virtual reality: challenges and research agenda. Journal of Big Data 2, 1 (01 Oct 2015), 22. 2015. https://doi.org/10.1186/s40537-015-0031-2
  23. Scott Riddle, Daniel Wasser, and Michael McCarthy. Touching The Human Neuron: User-Centric Augmented Reality Viewing and Interaction of in-vivo Cellular Confocal Laser Scanning Microscopy (CLSM) Utilizing High Resolution zStack Data Sets. Journal of Biocommunication 41, 1, 2017.
  24. Ronell Sicat, Jiabao Li, JunYoung Choi, Maxime Cordeil, Won-Ki Jeong, Benjamin Bach, and Hanspeter Pfister. DXR: A Toolkit for Building Immersive Data Visualizations. IEEE Transactions on Visualization and Computer Graphics. 2018.
  25. Mauricio Sousa, Daniel Mendes, Soraia Paulo, Nuno Matela, Joaquim Jorge, and Daniel Simoes Lopes. VRRRRoom: Virtual Reality for Radiologists in the Reading Room. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4057-4062. 2017.
  26. Daniele Sportillo, Alexis Paljic, Mehdi Boukhris, Philippe Fuchs, Luciano Ojeda, and Vincent Roussarie. An immersive Virtual Reality system for semiautonomous driving simulation: a comparison between realistic and 6-DoF controller-based interaction. In Proceedings of the 9th International Conference on Computer and Automation Engineering. ACM, 6-10. 2017.
  27. Will Usher, Pavol Klacansky, Frederick Federer, Peer-Timo Bremer, Aaron Knoll, Jeff Yarch, Alessandra Angelucci, and Valerio Pascucci. A virtual reality visualization tool for neuron tracing. IEEE transactions on visualization and computer graphics 24, 1. 994-1003. 2018. https://doi.org/10.1109/TVCG.2017.2744079
  28. Eric Whitmire, Hrvoje Benko, Christian Holz, Eyal Ofek, and Mike Sinclair. Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 86. 2018.
  29. Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, Tadao Ichikawa: The Go-Go Interaction Technique: Non-Linear Mapping for Direct Manipulation in VR. ACM Symposium on User Interface Software and Technology 79-80, 1996.
  30. Bowman, D. A., & Hodges, L. F. An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In Proceedings of the 1997 symposium on Interactive 3D graphics(pp. 35-ff). ACM. 1997.
  31. Google Tilt Brush. https://www.tiltbrush.com/.
  32. ARToolkit. www.artoolkit.org;
  33. Mixed Reality Toolkit. https://github.com/Microsoft/MixedRealityToolkitUnity
  34. D. Filonik, T. Bednarz, M. Rittenbruch, and M. Foth. Glance: Generalized geometric primitives and transformations for information visualization in AR/VR environments. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1, VRCAI '16, pp. 461-468. ACM, New York, NY, USA, 2016

Cited by

  1. Development of a Remote Augmented Reality Board Game Wearing HMD Based on Hand Motion Tracking vol.20, pp.12, 2019, https://doi.org/10.9728/dcs.2019.20.12.2583