Determinants of User Satisfaction with Mobile VR Headsets: The Human Factors Approach by the User Reviews Analysis and Product Lab Testing

Abstract

Since the VR market is expected to have a high growth, this study aimed to investigate the human factor-related determinants of user satisfaction with mobile VR headsets. A pre-study of customer reviews was conducted with the help of semantic network analysis to identify the core keywords for understanding negative and positive predictors of mobile VR headset experiences. Through laboratory testing with three different commercial models, the main study measured and identified the predictors of user satisfaction. From the results, five factors were extracted as valid predictor variables and used for regression analysis. These factors were immersion, VR sickness, usability, wear-ability and menu navigation interface. All the five predictors were proved to be significant determinants of the perceived user satisfaction with mobile VR headsets. Usability was the strongest predictor, followed by VR sickness and wear-ability. Practical and theoretical implications of the results were discussed.

1. INTRODUCTION 

Virtual reality (VR), an artificial environment that provides an immersive experience, is not a new technology. Since the first VR head-mounted display, SKETCHPAD, was developed by Ivan Sutherland in 1966[1], a variety of VR devices or virtual environment systems have been introduced. However, until recently, most VR systems had failed to attract attention from users in the consumer market because of the low performance of the hardware and software, extremely high price, and low wearability of the headsets. As a result of the improved performance of both hardware and software as well as the lower market price, VR entered an emerging market after 2010 [2]. Followed by Facebook’s acquisition of Oculus in 2014, major manufactures such as Google, Samsung, Microsoft, LG, HTC, SONY, and numerous minor developers have introduced VR headsets onto the market. Market research predicts that the VR market is expected to grow rapidly and become a 30–40 billion USD industry in 2020 and mobile VR/AR will be the primary driver for market expansion.[3, 4, 5]

It  was  approximately  two  decades  ago  when  a  few ergonomists called for a paradigm shift of future VR research. They brought attention to the human factors and usability issues  associated  with  VR  interface  design[6], [7].  They examined users’ perception of problems with the design of VR  devices  that  feature a head-mounted  display  and,  in consequence of discovering lack of satisfaction with VR use, proposed possible causes such as heavy weight, poor fit and adjustability of headsets, unusable design of input devices, and inflexibility of head movement due to cables. A decade later, Sharples et al [8]. again called attention to the issue of interrelations among key human factors such as input device usability,  screen menu interface  design,  ergonomics  of  a headset, VR sickness, and presence perception.

Transition from the lab prototype to the market product now  requires  a  new framework  of  research  agenda  and evaluation methods for quality of use (QoE) by customers of mobile VR devices. Thus, it is imperative that VR research be extended into the market research to question which factors are more important to the customers and how those factors are interrelated and balanced with each other. Also, it is an initiative that highlights the overall user satisfaction as an outcome rather than by measurements of task completion or  error  rates. The question  of what  are  the  contributing factors of user satisfaction with the commercial VR headset products remains largely untested. All human factors should be  considered altogether  in  order  to  apprehend  the  user’s point of view. Diverse  evaluation techniques  should  be considered so that both design and ergonomics practitioners and  marketing  managers  can promote  improvement and success of mobile VR devices.

This study aims to  investigate  the  most  popular  VR product in the market: mobile VR headsets of smartphone platforms. VR devices on the market are available on two different platforms: PC/consoles and smartphones. The VR headsets in mobile and PC/console platforms share common technical features such  as sensors and headset shape; however, there are differences in many customer  related features in terms of content and market accessibility. Mobile platform  headsets  are  cheaper than PC/console systems: Google Cardboard sells for just $10 and the most popular Samsung  Gear  VR  is  about  $100. Thus, mobile  platform headsets seem to be more accessible to consumers. While PC/console headsets are more oriented toward video game content, mobile platforms  with  open  application  market stores provide a wider range of content and a convenient download/purchase process. It is reported in recent user study that mobile-based VR systems did now show differences in the perception of presence, usability, sickness, and satisfaction from PC/console system [9]. Furthermore, as YouTube   already  launched the official Virtual Reality channel and other major online video service providers such as Netflix, Hulu, and HBO want to distribute their three-dimensional (3D) and VR movies, mobile VR can even be extended into the media industry.

However, mobile VR headsets have some disadvantages related mainly to human factors which have been reported in VR research for decades and have yet to be resolved: heavy weight, overheat from the phone, lack of controllers, poor resolution, difficult adjustment (e.g., interpapillary and lens– eye distances), and motion sickness [10]. Thus, mobile VR headsets  may  have  a  low  user  perception of  satisfaction, which may lead to negative experiences and discontinued use of the device.

This study focuses on the human factor-related determinants of user satisfaction for the mobile VR headset: What are the significant predictors of satisfactory VR experience and which is the most important factor? Television manufacturing industry has a disastrous history of wearable devices such  as 3D glasses. Many factors contributed to the failure of 3D glasses, but there was a lack of research on the human factors that caused the negative user satisfaction and experiences. Thus, this  study  can contribute to understanding the human factors that affect the user perception of mobile VR headsets.

 

2. PRE-STUDY AND RELATED CONCEPTS

Despite the rapid increase of commercial VR headset sales, most researchers interested in this domain still face a lot of difficulties with the research design for human factors. Despite  the  abundance  of  prototype  lab  testing  on  the technical factors, there has been a lack of empirical studies on the user factors that negatively or positively influence user experiences with the VR headset. Thus, before the design of the main study, an explorative pre-study was conducted to identify the possible key factors of VR experience.

Because most previous studies on VR systems were conducted in the lab, evaluation methods of system quality and user factors have been restricted to questionnaire, interview, behavioral or physiological responses from the lab testing and observation. It is a new era for VR researchers, who can now gather data from real users in a natural setting. For   example,   analysis   of   customer   reviews   has the advantages of gathering data from real users and reflecting the  issues  from long  time  use  experiences,  not  from  lab experiments.

 

2.1 Pre-study of User Reviews on VR Headsets

A semantic network analysis on customer reviews of VR headsets  was  conducted. As  an  opinion  mining  method, semantic network analysis has been widely used for online customer reviews, and it has the advantage of mapping issues from the users’ point of view.[11], [12], [13], [14]

User  comments  on  VR  headsets  posted  as  product reviews  on  Amazon,  Best Buy,  and  other  retailers  were gathered in 2016 and analyzed with the text mining solution. Omitting comments on price or short sentiments (e.g., “it is great,” “cool,” or “disappointing”), a total of 100 user review cases   that   mentioned   positive  or negative   experiences regarding their actual uses of the VR headset were collected. A total of 474 keywords were extracted from the collected user comments and 106 unique keywords were transformed into a network matrix for analysis. UCINET and NodeXL software were used for data processing and analysis. The top 20 keywords appearing  in  the  100  user  comments  are presented in Table 1.

 

Table 1. Top 20 keywords of user comments on VR headsets

 

Analyzing the linkage patterns among keywords that co-occurred in the same user comment, we identified a well-connected network and some sub-groups of keywords. Fig. 1 illustrates a map of the linkage patterns among 106 unique keywords. The top 10 keywords that had higher numbers of

 

2.2 Concepts Related to VR Use Experience

In consideration of the pre-study on the user reviews of VR  headset  experiences, the  relevant  concepts  that  are studied with human factors or interface issues were reviewed.

 

2.2.1 Immersion and presence

Every visual display system such as IMAX, 3D TV, and home theatres has been designed to provide users with more realistic, direct, immediate, and natural sensory  stimulus. Defined  as  the  cognitive belief of being there  or  the perception of the virtual environment as a real environment [15], [16], presence has been the core of VR experience [17], [18], [19]. Immersion is a perceived technological condition that leads to the cognitive (mis)belief of presence in the virtual environment by simulating sensory stimulus congruent with real world experiences [20]. Referred to as the technical level of sensory fidelity in a VR system [21], immersion is the foundation for positive VR experience [22].

Many factors have been known to be contributors to immersive VR experience: a wide field-of-view (FOV), lack of the screen door effect, a lack of latency between head tracking motion and screen transition, and a lack of light leakage. Both hardware (e.g., the size and thickness of the lens, resolution of the display, and headset attachment) and software (e.g., point of view, display lag, and surround sound) are determinants of the sense of  immersion  in  virtual environment systems [23].

 

2.2.2 VR sickness

The  most  serious  and  negative  human  factor  of  VR experience is cyber sickness or VR sickness[24], [25], [26]. It has been reported that more than 60% of first time users feel sick links were “comfortable” (63 links with other keywords), “easy” (33), “experience” (27), “immersive” (27), “phone” (25), “lens” (23), “touchpad” (23), “heavy/light” (20), “adjust” (19), and “screen” (17) [27]. This  causes  an  unpleasant  VR  use  experience  and reduces the use time [28].

Both the device and content factors contribute to the VR sickness. The theory of sensory conflict explains that VR sickness is caused by the mismatch between the user’s visual signals from the VR environment and the user’s expectations based on prior  experience  in  the  real  world [29].  Head tracking, a core motion function of VR, causes a slight time lag between head movement and visual image transition [30], [31]. This latency has been identified as an underlying trigger of  VR  sickness. Other contributors  to  VR  sickness  are  a narrow FOV, optical distortion, and motion-tracking mismatch [32], [33]. In addition, some visual effects in VR content such as fast cut editing, rapid camera rotations, and first person point-of-view have been identified as causes [34], [35].

 

2.2.3 Usability

Usability has been a fundamental human factor condition for satisfaction with most computer systems and consumer products. Given that a VR system requires user interactions with the touchpad, controller, and motion sensors, multiple usability dimensions may contribute to the quality perception of VR headsets [36]. Among these dimensions, intuitive, direct, and flexible manipulation of the physical or virtual interfaces of headsets may be the most significant in terms of ease of use. Especially in the VR context, a sense of user control, or controllability, can be a crucial dimension of usability because when users believe that they are actively controlling the environment, they may feel less VR sickness [37].

To resolve the issue of usability for VR headsets, most manufacturers   provide  out-of-box experience (OOBE) tutorials so that new users can learn how to control the device. However,  designing  easy,  flexible,  and  controllable  user interfaces  and interactions  is  definitely  not  an  easy  task because the diverse genres of VR content require different interface modes [38], [39]. Furthermore, a usability design is a more critical factor on a mobile VR platform than on a PC/console platform because mobile VR headsets face another contextual usability problem when incoming calls or messages are received during the use of the VR content.

 

2.2.4 Wearability

As a head-mounted wearable device, the VR headset has multiple ergonomic attributes  that  affect  the  negative  or positive  evaluation of a use experience [13]. The  most important form factor of a VR headset is weight. If users feel that the head-mounted display unit is too heavy, they are liable to lose the sense of immersion and become fatigued more quickly [40].

Along with the adjustable straps, the material and shape of the nose bridge can help to reduce the pressure on the nose and adjust weight distribution. However, if the attachment of the headset is loose, the head tracking motion may cause screen aliasing, loss of immersion, and VR sickness [41]. Because the facial structure of each user is slightly different, it is critical to provide comfort by finding a balance between a  gentle  fit  on  the  nose  bridge  and  the  tightness  of  the attachment.

The  distance  between  the  lenses  and  eyes  is  another ergonomic issue. While some mobile VR headsets do not provide IPD (inter-pupillary distance) adjustment, many of them have a control knob to adjust the distance of the lenses from the eyes for focusing. A larger distance provides a space for users with eye glasses, but detracts from the FOV, which affects the sense of immersion [42].

 

2.2.5 Navigation interface

For a user to start playing VR content, the navigation interface  should  be  properly designed  for  browsing  and selecting menus in the home or app market screen. However, very little research has been conducted on navigation design and its impact on VR experience. Similar to the design of information architecture in most digital entertainment systems, virtual environments also require universal interaction tasks such as wayfinding, travel, selection, and system control [43], [44].

Usually for the mobile HMD, the coordination between head movement and a virtual pointer is required for selection. Also, a combination of touchpad and buttons on the top or side of the headset is used for execution. Thus, the size and position of the virtual icons in a menu can decrease satisfaction with the VR experience because users may find it difficult to make a precise head movement and stabilize the head position when using their finger to control the touchpad and buttons [45].

 

3. MAIN STUDY: METHOD

3.1 Participants

Mobile phone users who were aware of and interested in VR were recruited for the lab testing in May 2016. A total of 30 people(22 males, 8 females) participated in the experiment, and the average age of the sample was 30.1 years (sd = 6.9).

 

3.2 Experiment apparatus

To represent the diversity of the VR headset’s specifications, we selected three mobile VR headsets available on the market: Samsung Gear VR, Baofeng Mojing 3 Plus, and LG 360 VR. These headsets suitably demonstrate the diversity of attributes in size, phone attachment type, weight, display (lens) size, IPD adjustments, pixel density, resolution, FOV, arm type, user interface, and retail price. Among these attributes, wearability may be impacted by the total  weight  (device weight  +  phone  weight),  arm  type (adjustable  headband  or  plastic  arms),  and type  of  nose bridge (padding or silicon). Immersion can be influenced by display attributes such as pixel density and resolution, FOV, screen clarity, focusing, and the distance between the eyes and lenses.

 

3.3 Measurement

After analyzing user comments and related literature, a total of 13 items were selected  as  possible  predictors  or determinants  of  satisfaction  with  the  mobile VR  headset experience (see Table 2).

Three  items  dealt  with  the  quality  of  the  screen  and display: width of angle, clarity, and smooth transition during head tracking. These items  are  also conditions for  the immersive perception of a virtual environment. From the 16 standardized Simulator  Sickness  Questionnaire  statements, the  most  prominent VR  sickness symptoms were chosen: nausea and eye strain [46]. Three usability items measured the ease of use, flexibility, and sense of self control. The perceived comfort of the system weight, nose bridge, and distance between the eyes and lenses were also evaluated by questions. Two items for navigation design were measured: the suitability of the size and position of the menu icons. The level of overall satisfaction with the VR experience was measured as a dependent variable. All items were measured using a seven-point Likert-type scale (1 = most disagree, 7 = most agree).

 

3.3.4 Procedure

Participants were invited to the testing lab and given an orientation session about the purpose and procedure of the testing by a moderator. Each participant had a preliminary session to practice wearing the headsets and adjusting the focus for clear viewing. In the main session, each participant was asked to wear the first headset, browse the main screen, activate a menu for a designated VR video, and experience the content for 10 minutes. We asked participants to use the VR in a seated position instead of standing because most mobile VR users at home use the device in a seated position.

 

Table 2. Measurement Items

 

Two  types  of  mobile  VR  content  were  presented  to participants: a roller coaster simulation and a driving game (See Fig.2). The roller coaster was selected to create a 3D video experience with a feeling of presence.  Little head tracking  was required for this simulation video, but we observed that most participants moved their heads around to browse the theme park environment in the video. The driving game was chosen for its interactive experience with a sense of control. A large amount of head tracking was required to control the car, which could cause VR sickness.

 

Fig. 1. Mapping of user comments keywords

Fig. 2. Screen Images of VR Content Used in the Testing (image sources: (Left) Dive City Rollercoaster and (Right) VR Car Racing 3D for Google Cardboard)

 

After the VR experience  session, the participant was asked to  remove  the headset and evaluate the degrees of sickness, immersion, usability, navigation, and satisfaction in the questionnaire for the testing device. After a five minute break, the session  was  repeated  for  the  second  and  third devices.  The  order of devices was  randomized  for  each participant. The total time for the whole testing session for each participant was about 75 minutes.

 

4. RESULTS

The  responses  of  the  30  participants  for  the  three devices were coded and a total of 90 cases were analyzed using  SPSS  version  21.  After  checking  the descriptive statistics,  a  factor  analysis  and  a  reliability  test  were conducted  to confirm  the  dimensionalities  among  the  13 independent variables and to reduce the number of predictors for  the  regression  analysis.  With  the  factors  identified, a multiple   regression  analysis  was conducted to confirm whether those factors were significant predictors and which factor was the most powerful determinant  of the  user perception of satisfaction with the VR headset.

 

4.1 Factor analysis

A  factor  analysis  with  a  direct  oblimin  rotation  was conducted for non-orthogonal solution [47]. A total of four factors  were  extracted  in  the  pattern matrix.  Usability, wearability, and navigation items were factored as predicted. However, three screen items and two VR sickness items were combined into a single factor. This is because of the high association between immersion and VR sickness. A greater amount of immersive stimulus a user has in the VR environment leads to a higher probability of the VR sickness symptoms.

 

Table 3. Descriptives of Three VR headsets (mean and s.d.)

 

These two concepts may share the same dimensionality, but there is an apparent distinction in conceptual dimensions. Thus, a further analysis was conducted. A series of reliability tests demonstrated that the original factor was better divided into two separate factors: immersion factor (Cronbach’s α = .68) and VR sickness (Cronbach’s α = .86).

 

Table 4. Results of factor analysis and reliability test

 

4.2 Correlation: Interrelations between factors

In order to examine interrelations between the factors, a correlation  analysis  was conducted. All causative factors were highly correlated with satisfaction. Very strong associations were found between VR sickness and immersion (r=.60, p<.001), and between VR sickness and wearability (r=.55,  p<.001). Navigation  of menu interface was also highly correlated with usability (r=.51, p<.001) and immersion (r=.42, p<.001). Immersion had strong relations with usability (r=.33, p<.01) and wearability(r=.33, p<.01). Mild associations were found between wearability and usability (r=.25, p<.05), and between wearability and navigation (r=.22,  p<.05). No significant correlation was found between VR sickness and navigation.

 

Table 5. Inter-correlations between factors

 

4.3 Regression: Determinants of VR headset satisfaction

A multiple regression with stepwise selection method was conducted on the satisfaction with the VR headset for five predictor factors:  immersion,  VR sickness, usability, wearability, and navigation. The results show that the regression model was very strong (R²= .70, F = 42.86, p < .001) and all five predictors were statistically significant. All VIF scores were lower than 10 (range: 1.42–2.08); thus, there was no multicollinearity problem.

The  strongest  predictor  of  VR  headset  satisfaction  is usability (β = .29, p < .001), followed very closely by VR sickness (β = .28, p < .01), wearability (β = .24, p < .01), and immersion (β = .20, p < .01). Navigation is the least strong predictor of satisfaction (β = .16, p < .05).

 

5. DISCUSSIONS AND CONCLUSION

5.1 Summary and Discussions

This study aimed to investigate human factor-related determinants of user satisfaction with mobile VR headsets. As the VR market is expected to have high growth, the user perception of VR headset quality has become an important research topic both for engineers and designers. Given the lack of past research on this agenda, a pre-study on VR headset user reviews was conducted and core keywords for understanding negative and positive attributes that may affect the user  experience with the mobile VR  systems  on  the market were found.

 

Table 6. Result of regression analysis on satisfaction with the VR headsets

 

Through lab testing with three different models for 30 participants, the main study measured 13 predictors of user satisfaction. Five factors were extracted as predictors and used for regression analysis: immersion, VR sickness, usability, wearability, and navigation. All five predictors were proved to be significant determinants of perceived user satisfaction with mobile  VR headsets. Usability was  the strongest predictor, followed by VR sickness and wearability.  

The results of this study pose several issues for further discussion. First, many manufacturers have highlighted the improvements in technical specifications for immersive VR experience such as screen resolution, refresh rate, lack of light leakage, and wide FOV. However, ease of use, a sense of self-control, and flexibility, which are common usability conditions for most human–computer interactions, are still more important human factors than the technical specifications. Specifically, from the standpoint of product user interface design, an ergonomic approach should be considered regarding the position and type of physical interfaces for menu selection and control. From the standpoint of graphic user interface design, clearer legibility and less depth should be a design guideline for VR usability.  

Second, VR sickness is still a major detractor of user satisfaction. The degree of VR sickness symptoms is highly associated with the level of immersion. Thus, properly balancing the sensory stimulus level for immersive but also sustainable experience is needed for device and software design. Given that most VR manufacturers currently recommend a break after 30 minutes of use, the question of “how much immersion is enough” [48] should be investigated under various conditions for the headset type, genre of content, interface style, and user characteristics.

Third, as illustrated in the pre-study, adjectives such as “easy,” “immersive,” “heavy,” “fun,” and “comfortable” were the most frequent and the most linked keywords in the user reviews. The feeling of comfort was associated with many VR  factor  dimensions:  wearability  (e.g.,  “wear,”  “strap,” “focus,” “insert tray,” and “phone”), usability (“easy,” “use,” and “control”), navigation (“interface,” “navigate,” and “button”),  and  immersion  (“screen” and  “vision”).  Thus, comfort should be paid more attention as both a key affective and  ergonomic  value  for the VR experience. Thus, it is recommended  that  VR  headset  designers  consider weight, band attachment type and padding material primarily for the comfort of the users.

Lastly,   interface   design   for   menu   navigation   was identified as a significant factor for optimal VR experience. Because the beginning phase of mobile VR use includes user actions for browsing and selecting applications and settings, a well-structured menu in the virtual screen that utilizes head tracking and a physical touchpad in the headset for finger taps is needed. As the VR ecosystem evolves into a combined platform with the smartphone, more users will purchase VR content through the app store while wearing a VR headset. Thus, future studies are needed on VR-friendly information architecture and navigation interfaces.

 

5.2 Conclusion

The  main  contribution  of  this  study  lies  in  the  user-oriented and holistic approach to the understanding of the VR experience on the mobile platform headsets. Previous studies on VR environments have mostly focused on technical or cognitive  aspects  of  VR  system  use on PC  platforms.  In addition, many previous studies have investigated each part of the VR experiences separately. For example, VR sickness, immersion, usability, wearability, and menu interface design were not explored in tandem.

The academic contribution of this study is that it may be the first introduction of a human–computer interaction-based research model of VR satisfaction with multiple predicting factors. Clearly, more empirical testing, consisting of more samples and a diversity of VR headset models, is needed to develop this model. Its practical contribution is the empirical support for the coordination between ergonomics engineers and   interaction   designers   for   successful   VR   product development.

The small sample size was a limitation of this study. Because the testing for each participant required the difficult task of wearing multiple headsets for more than an hour, the recruitment of participants was rather difficult and the age and gender distribution of the sample was quite narrow. With the rollercoaster and racing car simulations, the testing of three models of this study showed rather wide differences in the perception of headset weight, wearability, and the ease of navigation interface. Thus, future studies should consider the representativeness of testers and diversity of the VR contents and headset models.

Another suggestion for future research is to consider the possible interaction effect between the headset device design and content design. For example, immersion and VR sickness are caused by head tracking, which are both affected by the sensitivity of sensors in the headset and by the point-of-view and motion direction of a user character in the VR environment.