Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Survey of Multimodal Systems and Techniques for Motor Learning

  • Tadayon, Ramin (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University) ;
  • McDaniel, Troy (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University) ;
  • Panchanathan, Sethuraman (Center for Cognitive Ubiquitous Computing, School of Computing, Informatics, and Decision Systems Engineering, Arizona State University)
  • Received : 2016.12.24
  • Accepted : 2016.12.30
  • Published : 2017.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.

Keywords

References

  1. H. M. Huang, U. Rauch, and S. S. Liaw, "Investigating learners attitudes toward virtual reality learning environments: based on a constructivist approach," Computers & Education, vol. 55, no. 3, pp. 1171-1182, 2010. https://doi.org/10.1016/j.compedu.2010.05.014
  2. M. K. Holden, "Virtual environments for motor rehabilitation: review," CyberPsychology & Behavior, vol. 8, no. 3, pp. 187-211, 2005. https://doi.org/10.1089/cpb.2005.8.187
  3. B. Dalgarno and M. J. Lee, "What are the learning affordances of 3-D virtual environments?," British Journal of Educational Technology, vol. 41, no. 1, pp. 10-32, 2010. https://doi.org/10.1111/j.1467-8535.2009.01038.x
  4. G. Saposnik and M. Levin, "Virtual reality in stroke rehabilitation a meta-analysis and implications for clinicians," Stroke, vol. 42, no. 5, pp. 1380-1386, 2011. https://doi.org/10.1161/STROKEAHA.110.605451
  5. D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V. Adamovich, M. Recce, and H. Poizner, "Virtual reality-enhanced stroke rehabilitation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 9, no. 3, pp. 308-318, 2001. https://doi.org/10.1109/7333.948460
  6. A. S. Merians, D. Jack, R. Boian, M. Tremaine, G. C. Burdea, S. V. Adamovich, M. Recce, and H. Poizner, "Virtual reality-augmented rehabilitation for patients following stroke," Physical Therapy, vol. 82, no. 9, pp. 898-915, 2002.
  7. M. S. Cameirao, S. B. I. Badia, L. Zimmerli, E. Duarte Oller, and P. F. M. J. Verschure, "The rehabilitation gaming system: a virtual reality based system for the evaluation and rehabilitation of motor deficits," in Proceedings of the 6th 2007 International Workshop on Virtual Rehabilitation, Venice, Italy, 2007, pp. 29-33.
  8. M. S. Cameirao, S. B. I. Badia, E. Duarte Oller, and P. F. M. J. Verschure, "The rehabilitation gaming system: a review," Studies in Health Technology and Informatics, vol. 145, pp. 65-83, 2009.
  9. A. Hayes, P. Dukes, and L. F. Hodges, "A virtual environment for post-stroke motor rehabilitation," 2012 [Online]. Available: https://www.cs.clemson.edu/nsfreu/documents/stroke_rehab_abstract.pdf.
  10. M. Mihelj, D. Novak, M. Milavec, J. Ziherl, A. Olensek, and M. Munih, "Virtual rehabilitation environment using principles of intrinsic motivation and game design," Presence, vol. 21, no. 1, pp. 1-15, 2012. https://doi.org/10.1162/PRES_a_00078
  11. A. Rizzo and G. J. Kim, "A SWOT analysis of the field of virtual reality rehabilitation and therapy," Presence, vol. 14, no. 2, pp. 119-146, 2005. https://doi.org/10.1162/1054746053967094
  12. K. Laver, S. George, S. Thomas, J. E. Deutsch, and M. Crotty, "Virtual reality for stroke rehabilitation," Stroke, vol. 43, no. 2, pp. e20-e21, 2012. https://doi.org/10.1161/STROKEAHA.111.642439
  13. B. Lange, S. M. Flynn, and A. A. Rizzo, "Game-based telerehabilitation," European Journal of Physical and Rehabilitation Medicine, vol. 45, no. 1, pp. 143-151, 2009.
  14. C. Kirner and T. G. Kirner, "Development of an interactive artifact for cognitive rehabilitation based on augmented reality," in Proceedings of 2011 IEEE International Conference on Virtual Rehabilitation (ICVR), Zurich, Switzerland, 2011, pp. 1-7.
  15. Y. M. Aung and A. Al-Jumaily, "AR based upper limb rehabilitation system," in Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, 2012, pp. 213-218.
  16. J. C. Perry, H. Zabaleta, A. Belloso, C. Rodriguez-de Pablo, F. I. Cavallaro, and T. Keller, "ArmAssist: development of a functional prototype for at-home telerehabilitation of post-stroke arm impairment," in Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, 2012, pp. 1561-1566.
  17. H. Kazemi, R. E. Kearney, and T. E. Milner, "Using a robotic interface and haptic feedback to improve grip coordination of hand function following stroke: case study," in Proceedings of 2013 IEEE International Conference on Virtual Rehabilitation (ICVR), Philadelphia, PA, 2013, pp. 196-197.
  18. C. Rodriguez-de-Pablo, J. C. Perry, F. I. Cavallaro, H. Zabaleta, and T. Keller, "Development of computer games for assessment and training in post-stroke arm telerehabilitation," in Proceedings of 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, 2012, pp. 4571-4574.
  19. M. Ma and K. Bechkoum, "Serious games for movement therapy after stroke," in Proceedings of 2008 IEEE International Conference on Systems, Man and Cybernetics, Singapore, 2008, pp. 1872-1877.
  20. M. J. Johnson, M. Trickey, E. Brauer, and X. Feng, "TheraDrive: a new stroke therapy concept for home-based, computer-assisted motivating rehabilitation," in Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, 2004, pp. 4844-4847.
  21. R. Ruparel, M. J. Johnson, E. Strachota, J. McGuire, and G. Tchekanov, "Evaluation of the TheraDrive system for robot/computer assisted motivating rehabilitation after stroke," in Proceedings of 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, 2009, pp. 811-814.
  22. G. Alankus, R. Proffitt, C. Kelleher, and J. Engsberg, "Stroke therapy through motion-based games: a case study," in Proceedings of 12th International ACM SIGACCESS Conference on Computers and Accessibility, Orlando, FL, 2010, pp. 219-226.
  23. A. Jacobs, A. Timmermans, M. Michielsen, M. Vander Plaetse, and P. Markopoulos, "CONTRAST: gamification of arm-hand training for stroke survivors," in Proceedings of the 31st Annual CHI Conference on Human Factors in Computing Systems Presence Abstract (CHI'13), Paris, 2013, pp. 415-420.
  24. G. Alankus and C. Kelleher, "Reducing compensatory motions in video games for stroke rehabilitation," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'12), Austin, TX, 2012, pp. 2049-2058.
  25. R. Garris, R. Ahlers, and J. E. Driskell, "Games, motivation, and learning: a research and practice model," Simulation & Gaming, vol. 33, no. 4, pp. 441-467, 2002. https://doi.org/10.1177/1046878102238607
  26. K. Salen and E. Zimmerman, "Game design and meaningful play," in Handbook of Computer Game Studies. Cambridge, MA: MIT Press, 2005, pp. 59-79.
  27. J. W. Burke, M. D. J. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie, and S. M. McDonough, "Optimising engagement for stroke rehabilitation using serious games," The Visual Computer, vol. 25, no. 12, pp. 1085-1099, 2009. https://doi.org/10.1007/s00371-009-0387-4
  28. D. Johnson and J. Wiles, "Effective affective user interface design in games," Ergonomics, vol. 46, no. 13-14, pp. 1332-1345, 2003. https://doi.org/10.1080/00140130310001610865
  29. J. Wiemeyer and A. Kliem, "Serious games in prevention and rehabilitation: a new panacea for elderly people?," European Review of Aging and Physical Activity, vol. 9, no. 1, pp. 41-50, 2012. https://doi.org/10.1007/s11556-011-0093-x
  30. K. Ogomori, M. Nagamachi, K. Ishihara, S. Ishihara, and M. Kohchi, "Requirements for a cognitive training game for elderly or disabled people," in Proceedings of 2011 International Conference on Biometrics and Kansei Engineering, Takamatsu, Japan, 2011, pp. 150-154.
  31. E. Flores, G. Tobon, E. Cavallaro, F. I. Cavallaro, J. C. Perry, and T. Keller, "Improving patient motivation in game development for motor deficit rehabilitation," in Proceedings of 2008 International Conference on Advances in Computer Entertainment Technology, Yokohama, Japan, 2008, pp. 381-384.
  32. P. Rego, P. M. Moreira, and L. P. Reis, "Serious games for rehabilitation: a survey and a classification towards a taxonomy," in Proceedings of 5th Iberian Conference on Information Systems and Technologies, Santiago de Compostela, Spain, 2010, pp. 1-6.
  33. G. Alankus, A. Lazar, M. May, and C. Kelleher, "Towards customizable games for stroke rehabilitation," in Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI'10), Atlanta, GA, 2010, pp. 2113-2122.
  34. J. L. Gonzalez Sanchez, N. Padilla Zea, and F. Gutierrez, "From usability to playability: introduction to playercentred video game development process," in Proceedings of 1st International Conference on Human Centered Design, San Diego, CA, 2009, pp. 65-74.
  35. J. W. Burke, M. McNeill, D. Charles, P. Morrow, J. Crosbie, and S. McDonough, "Serious games for upper limb rehabilitation following stroke," in Proceedings of 2009 Conference in Games and Virtual Worlds for Serious Applications, Coventry, UK, 2009, pp. 103-110.
  36. A. Gouaich, N. Hocine, L. Van Dokkum, and D. Mottet, "Digital-pheromone based difficulty adaptation in post-stroke therapeutic games," in Proceedings of 2nd ACM SIGHIT International Health Informatics Symposium, Miami, FL, 2012, pp. 5-12.
  37. J. Chen, "Flow in games (and everything else)," Communications of the ACM, vol. 50, no. 4, pp. 31-34, 2007. https://doi.org/10.1145/1232743.1232769
  38. B. Kim, H. Park, and Y. Baek, "Not just fun, but serious strategies: using meta-cognitive strategies in game-based learning," Computers & Education, vol. 52, no. 4, pp. 800-810, 2009. https://doi.org/10.1016/j.compedu.2008.12.004
  39. B. Paras, "Game, motivation, and effective learning: an integrated model for educational game design," in Proceedings of 2005 Digital Games Research Association's 2nd International Conference: Changing Views: Worlds in Play, Vancouver, Canada, 2005.
  40. E. Hudlicka, "Affective game engines: motivation and requirements," in Proceedings of 4th International Conference on Foundations of Digital Games (FDG'09), Orlando, FL, 2009, pp. 299-306.
  41. K. Tanaka, J. R. Parker, G. Baradoy, D. Sheehan, J. R. Holash, and L. Katz, "A comparison of exergaming interfaces for use in rehabilitation programs and research," Loading..., vol. 6, no. 9, pp. 69-81, 2012.
  42. G. Saposnik, R. Teasell, M. Mamdani, J. Hall, W. McIlroy, D. Cheung, K. E. Thorpe, L. G. Cohen, and M. Bayley, "Effectiveness of virtual reality using wii gaming technology in stroke rehabilitation a pilot randomized clinical trial and proof of principle," Stroke, vol. 41, no. 7, pp. 1477-1484, 2010. https://doi.org/10.1161/STROKEAHA.110.584979
  43. M. R. Mouawad, C. G. Doust, M. D. Max, and P. A. McNulty, "Wii-based movement therapy to promote improved upper extremity function post-stroke: a pilot study," Journal of Rehabilitation Medicine, vol. 43, no. 6, pp. 527-533, 2011. https://doi.org/10.2340/16501977-0816
  44. B. Lange, C. Y. Chang, E. Suma, B. Newman, A. S. Rizzo and M. Bolas, "Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor," in Proceedings of 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, 2011, pp. 1831-1834.
  45. K. LaBelle, "Evaluation of Kinect joint tracking for clinical and in-home stroke rehabilitation tools," Ph.D. dissertation, University of Notre Dame, IN, 2011.
  46. Y. J. Chang, S. F. Chen, and J. D. Huang, "A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities," Research in Developmental Disabilities, vol. 32, no. 6, pp. 2566-2570, 2011. https://doi.org/10.1016/j.ridd.2011.07.002
  47. C. Y. Chang, B. Lange, M. Zhang, S. Koenig, P. Requejo, N. Somboon, A. A. Sawchuk, and A. A. Rizzo, "Towards pervasive physical rehabilitation using Microsoft Kinect," in Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, San Diego, CA, 2012, pp. 159-162.
  48. J. E. Deutsch, M. Borbely, J. Filler, K. Huhn, and P. Guarrera-Bowlby, "Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy," Physical Therapy, vol. 88, no. 10, pp. 1196-1207, 2008. https://doi.org/10.2522/ptj.20080062
  49. S. Brosnan, "The potential of Wii-rehabilitation for persons recovering from acute stroke," Physical Disabilities Special Interest Section Quarterly, vol. 32, no. 1, pp. 1-3, 2009.
  50. A. Hartveld and J. R. Hegarty, "Augmented feedback and physiotherapy practice," Physiotherapy, vol. 82, no. 8, pp. 480-490, 1996. https://doi.org/10.1016/S0031-9406(05)66414-0
  51. B. Bongers and S. Smith, "Interactivating rehabilitation through active multimodal feedback and guidance," in Smart Health Applications and Services: Developments and Practices. Hershey, PA: IGI-Global, 2011, pp. 236-260.
  52. L. Beursgens, A. Timmermans, and P. Markopoulos, "Playful arm hand training after stroke," in Proceedings of 2012 ACM Annual Conference on Human Factors in Computing Systems Extended Abstracts (CHI'12), Austin, TX, 2012, pp. 2399-2404.
  53. R. Sigrist, G. Rauter, L. Marchal-Crespo, R. Riener, and P. Wolf, "Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning," Experimental Brain Research, vol. 233, no. 3, pp. 909-925, 2015. https://doi.org/10.1007/s00221-014-4167-7
  54. J. Parker, S. Mawson, G. Mountain, N. Nasr, R. Davies, and H. Zheng, "The provision of feedback through computer-based technology to promote self-managed post-stroke rehabilitation in the home," Disability and Rehabilitation: Assistive Technology, vol. 9, no. 6, pp. 529-538, 2014. https://doi.org/10.3109/17483107.2013.845611
  55. J. Parker, S. Mawson, G. Mountain, N. Nasr, and H. Zheng, "Stroke patients utilisation of extrinsic feedback from computer-based technology in the home: a multiple case study realistic evaluation," BMC Medical Informatics and Decision Making, vol. 14, article no. 46, pp. 1-13, 2014. https://doi.org/10.1186/1472-6947-14-1
  56. J. Parker, G. Mountain, and J. Hammerton, "A review of the evidence underpinning the use of visual and auditory feedback for computer technology in post-stroke upper-limb rehabilitation," Disability and Rehabilitation: Assistive Technology, vol. 6, no. 6, pp. 465-472, 2011. https://doi.org/10.3109/17483107.2011.556209
  57. R. Sigrist, G. Rauter, R. Riener, and P. Wolf, "Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review," Psychonomic Bulletin & Review, vol. 20, no. 1, pp. 21-53, 2013. https://doi.org/10.3758/s13423-012-0333-8
  58. A. Godbout and J. E. Boyd, "Corrective sonic feedback for speed skating: a case study," in Proceedings of 16th International Conference on Auditory Display, Washington, DC, 2010, pp. 23-30.
  59. F. van Wijck, D. Knox, C. Dodds, G. Cassidy, G. Alexander, and R. MacDonald, "Making music after stroke: using musical activities to enhance arm function," Annals of the New York Academy of Sciences, vol. 1252, no. 1, pp. 305-311, 2012. https://doi.org/10.1111/j.1749-6632.2011.06403.x
  60. I. Wallis, T. Ingalls, T. Rikakis, L. Olsen, Y. Chen, W. Xu, and H. Sundaram, "Real-time sonification of movement for an immersive stroke rehabilitation environment," in Proceedings of 13th International Conference on Auditory Display, Montreal, Canada, 2007, pp. 26-29.
  61. R. Ronsse, V. Puttemans, J. P. Coxon, D. J. Goble, J. Wagemans, N. Wenderoth, and S. P. Swinnen, "Motor learning with augmented feedback: modality-dependent behavioral and neural consequences," Cerebral Cortex, vol. 21, no. 6, pp. 1283-1294, 2011. https://doi.org/10.1093/cercor/bhq209
  62. A. Alamri, R. Iglesias, M. Eid, A. El Saddik, S. Shirmohammadi, and E. Lemaire, "Haptic exercises for measuring improvement of post-stroke rehabilitation patients," in Proceedings of 2007 IEEE International Workshop on Medical Measurement and Applications, Warsaw, Poland, 2007, pp. 1-6.
  63. M. H. Milot, L. Marchal-Crespo, C. S. Green, S. C. Cramer, and D. J. Reinkensmeyer, "Comparison of error-amplification and haptic guidance training techniques for learning of a timing-based motor task by healthy individuals," Experimental Brain Research, vol. 201, no. 2, pp. 119-131, 2010. https://doi.org/10.1007/s00221-009-2014-z
  64. M. Mihelj, T. Nef, and R. Riener, "A novel paradigm for patient-cooperative control of upper-limb rehabilitation robots," Advanced Robotics, vol. 21, no. 8, pp. 843-867, 2007. https://doi.org/10.1163/156855307780851975
  65. D. Feygin, M. Keehner, and R. Tendick, "Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill," in Proceedings of 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS2002), Orlando, FL, 2002, pp. 40-47.
  66. G. Grindlay, "Haptic guidance benefits musical motor learning," in Proceedings of 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Reno, NE, 2008, pp. 397-404.
  67. M. C. Rhoads, G. B. Da Matta, N. Larson, and S. Pulos, "A meta-analysis of visual feedback for motor learning," Athletic Insight, vol. 6, no. 1, pp. 17-33, 2014.
  68. S. Moya, S. Grau, D. Tost, R. Campeny and M. Ruiz, "Animation of 3D Avatars for rehabilitation of the upper limbs," in Proceedings of 2011 3rd International Conference on Games and Virtual Worlds for Serious Applications, Athens, Greece, 2011, pp. 168-171.
  69. Z. Ruttkay and H. van Welbergen, "Elbows higher! Performing, observing and correcting exercises by a virtual trainer," in Proceedings of 8th International Conference on Intelligent Virtual Agents, Tokyo, Japan, 2008, pp. 409-416.
  70. J. T. Ramsburg, K. Bharadwaj, S. Conner, N. Nandakumar, and A. Simes, "Augmented motor skills learning," 2015 [Online]. Available: https://www.researchgate.net/publication/281035957_Augmented_Motor_Skills_Learning.
  71. H. T. Jung, T. Takahashi, Y. K. Choe, J. Baird, T. Foster, and R. A. Grupen, "Towards extended virtual presence of the therapist in stroke rehabilitation," in Proceedings of 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), Seattle, WA, 2013, pp. 1-6.
  72. D. L. Eaves, G. Breslin, and P. van Schaik, "The short-term effects of real-time virtual reality feedback on motor learning in dance," Presence, vol. 20, no. 1, pp. 62-77, 2011. https://doi.org/10.1162/pres_a_00035
  73. A. Jaume-i-Capo, P. Martinez-Bueso, B. Moya-Alcover, and J. Varona, "Improving vision-based motor rehabilitation interactive systems for users with disabilities using mirror feedback," The Scientific World Journal, vol. 2014, article no. 964576, 2014.
  74. H. Bunt, R. J. Beun, and T. Borghuis, Multimodal Human-Computer Communication: Systems, Techniques, and Experiments. Heidelberg: Springer, 1998.
  75. F. Quek, D. McNeill, R. Bryll, S. Duncan, X. F. Ma, C. Kirbas, K. E. McCullough, and R. Ansari, "Multimodal human discourse: gesture and speech," ACM Transactions on Computer-Human Interaction (TOCHI), vol. 9, no. 3, pp. 171-193, 2002. https://doi.org/10.1145/568513.568514
  76. R. Tadayon, T. McDaniel, M. Goldberg, P. M. Robles-Franco, J. Zia, M. Laff, M. Geng, and S. Panchanathan, "Interactive motor learning with the autonomous training assistant: a case study," in Proceedings of 17th International Conference on Human-Computer Interaction, Los Angeles, CA, 2015, pp. 495-506.
  77. M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith, "Unification-based multimodal integration," in Proceedings of 8th Conference on European Chapter of the Association for Computational Linguistics, Madrid, Spain, 1997, pp. 281-288.
  78. L. Wu, S. L. Oviatt, and P. R. Cohen, "Multimodal integration-a statistical view," IEEE Transactions on Multimedia, vol. 1, no. 4, pp. 334-341, 1999. https://doi.org/10.1109/6046.807953
  79. J. Y. Chai, P. Hong, and M. X. Zhou, "A probabilistic approach to reference resolution in multimodal user interfaces," in Proceedings of 9th International Conference on Intelligent User Interfaces, Funchal, Portugal, 2004, pp. 70-77.
  80. H. Mendonca, J. Y. L. Lawson, O. Vybornova, B. Macq, and J. Vanderdonckt, "A fusion framework for multimodal interactive applications," in Proceedings of 2009 International Conference on Multimodal Interfaces, Cambridge, MA, 2009, pp. 161-168.
  81. Y. Song, L. P. Morency, and R. Davis, "Multimodal human behavior analysis: learning correlation and interaction across modalities," in Proceedings of 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, 2012, pp. 27-30.
  82. L. Nigay and J. Coutaz, "A design space for multimodal systems: concurrent processing and data fusion," in Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, 1993, pp. 172-178.
  83. R. S. Schaefer, "Auditory rhythmic cueing in movement rehabilitation: findings and possible mechanisms," Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 369, article no. 20130402, pp. 1-9, 2014.
  84. S. Holland, R. L. Wright, A. Wing, T. Crevoisier, O. Hodl, and M. Canelli, "A pilot study using tactile cueing for gait rehabilitation following stroke," in Proceedings of 2nd International Workshop on ICTs for Improving Patients Rehabilitation Research Techniques, Oldenburg, Germany, 2014, pp. 222-233.
  85. R. Sigrist, G. Rauter, R. Riener, and P. Wolf, "Terminal feedback outperforms concurrent visual, auditory, and haptic feedback in learning a complex rowing-type task," Journal of Motor Behavior, vol. 45, no. 6, pp. 455-472, 2013. https://doi.org/10.1080/00222895.2013.826169
  86. H. B. Helbig and M. O. Ernst, "Visual-haptic cue weighting is independent of modality-specific attention," Journal of Vision, vol. 8, no. 1, pp. 21-21, 2008.
  87. H. S. Vitense, J. A. Jacko, and V. K. Emery, "Multimodal feedback: an assessment of performance and mental workload," Ergonomics, vol. 46, no. 1-3, pp. 68-87, 2003. https://doi.org/10.1080/00140130303534
  88. S. Oviatt, R. Coulston, and R. Lunsford, "When do we interact multimodally?: cognitive load and multimodal communication patterns," in Proceedings of 6th International Conference on Multimodal Interfaces, State College, PA, 2004, pp. 129-136.
  89. C. Schonauer, K. Fukushi, A. Olwal, H. Kaufmann, and R. Raskar, "Multimodal motion guidance: techniques for adaptive and dynamic feedback," in Proceedings of 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, 2012, pp. 133-140.
  90. B. Xiao, R. Lunsford, R. Coulston, M. Wesson, and S. Oviatt, "Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences," in Proceedings of 5th International Conference on Multimodal Interfaces, Vancouver, Canada, 2003, pp. 265-272.
  91. M. Sun, X. Ren, and X. Cao, "Effects of multimodal error feedback on human performance in steering tasks," Information and Media Technologies, vol. 6, no. 1, pp. 193-201, 2011.
  92. R. G. Carson and J. S. Kelso, "Governing coordination: behavioural principles and neural correlates," Experimental Brain Research, vol. 154, no. 3, pp. 267-274, 2004. https://doi.org/10.1007/s00221-003-1726-8
  93. C. D. Wickens, "Multiple resources and performance prediction," Theoretical Issues in Ergonomics Science, vol. 3, no. 2, pp. 159-177, 2002. https://doi.org/10.1080/14639220210123806
  94. J. L. Burke, M. S. Prewett, A. A. Gray, L. Yang, F. R. Stilson, M. D. Coovert, L. R. Elliot, and E. Redden, "Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis," in Proceedings of 8th International Conference on Multimodal Interfaces, Banff, Canada, 2006, pp. 108-117.
  95. M. A. Guadagnoli and T. D. Lee, "Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning," Journal of Motor Behavior, vol. 36, no. 2, pp. 212-224, 2004. https://doi.org/10.3200/JMBR.36.2.212-224
  96. F. Chen, N. Ruiz, E. Choi, J. Epps, M. A. Khawaja, R. Taib, B. Yin, and Y. Wang, "Multimodal behavior and interaction as indicators of cognitive load," ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 2, no. 4, article no. 22, 2012.
  97. C. H. Anderson, D. C. van Essen, and B. A. Olshausen, "Directed visual attention and the dynamic control of information flow," in Neurobiology of Attention. Boston, MA: Elsevier Academic Press, 2005, pp. 11-17.
  98. B. F. Miller and C. B. Keane, Encyclopedia and Dictionary of Medicine, Nursing, and Allied Health. Philadelphia, PA: W.B. Saunders, 1983.
  99. M. S. Sanders and E. J. McCormick, Human Factors in Engineering and Design, 6th ed. New York, NY: McGraw-Hill, 1987.
  100. D. C. Ruspini, K. Kolarov, and O. Khatib, "The haptic display of complex graphical environments," in Proceedings of 24th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 1997, pp. 345-352.
  101. M. O. Belardinelli, S. Federici, F. Delogu, and M. Palmiero, "Sonification of spatial information: audio-tactile exploration strategies by normal and blind subjects," in Proceedings of 13th International Conference on Universal Access in Human-Computer Interaction, San Diego, CA, 2009, pp. 557-563.
  102. S. J. Lederman and R. L. Klatzky, "Haptic perception: a tutorial," Attention, Perception, & Psychophysics, vol. 71, no. 7, pp. 1439-1459, 2009. https://doi.org/10.3758/APP.71.7.1439
  103. H. Z. Tan, C. M. Reed, and N. I. Durlach, "Optimum information transfer rates for communication through haptic and other sensory modalities," IEEE Transactions on Haptics, vol. 3, no. 2, pp. 98-108, 2010. https://doi.org/10.1109/TOH.2009.46
  104. M. Slater, "How colorful was your day? Why questionnaires cannot assess presence in virtual environments," Presence, vol. 13, no. 4, pp. 484-493, 2004. https://doi.org/10.1162/1054746041944849
  105. E. A. Alluisi, P. F. Muller, and P. M. Fitts, "An information analysis of verbal and motor responses in a forced-paced serial task," Journal of Experimental Psychology, vol. 53, no. 3, pp. 153-158, 1957. https://doi.org/10.1037/h0046294
  106. H. Z. Tan, N. I. Durlach, C. M. Reed, and W. M. Rabinowitz, "Information transmission with a multifinger tactual display," Perception & Psychophysics, vol. 61, no. 6, pp. 993-1008, 1999. https://doi.org/10.3758/BF03207608