Evaluating the Comfort Experience of a Head-Mounted Display with the Delphi Methodology

Abstract

This study developed evaluation indicators for the comfort experience of virtual reality (VR) headsets by classifying, defining, and weighting cybersickness-causing factors using the Delphi research method and analytic hierarchical process (AHP) approach. Four surveys were conducted with 20 experts on VR motion sickness. The expert surveys involved the 1) classification and definition of cybersickness-causing dimensions, classification of sub-factors for each dimension, and selection of evaluation indicators, 2) self-reassessment of the results of each step, 3) validity revaluation, and 4) final weighting calculation. Based on the surveys, the evaluation indicators for the comfort experience of VR headsets were classified into eight sub-factors: field of view (FoV)-device FoV, latency-device latency, framerate-device framerate, V-sync-device V-sync, rig-camera angle view, rig-no-parallax point, resolution-device resolution, and resolution-pixels per inch (PPI). A total of six dimensions and eight sub-factors were identified; sub-factor-based evaluation indicators were also developed.

1. Introduction

With the development of hardware and software and the widespread use of digital devices, the virtual reality (VR) device market continues to grow. The recent commercialization and increased availability of VR headsets, also referred to as head-mounted displays (HMDs), has sparked interest in the use of VR technology for research, military, medical, educational, and entertainment purposes [1]. experts predict that VR technology will change the way of life in the future, including how we work, entertain ourselves, communicate, and learn [2].

While the maturity and economic viability of VR technology are highly regarded with the emergence of various products and manufacturers over the past five years (the VR "second wave"), it is necessary to address certain aspects for increasing user adoption, such as user experience, usability, accessibility, and health effects [2]. cybersickness, a human factor affecting user experience, is of significant concern [3][4]. Cybersickness is a side effect of exposure to the virtual environment and is characterized by unpleasant physiological symptoms such as nausea and dizziness. Many studies have been aimed at uncovering the cause ofcybersickness, which is considered to be strongly related to the level of user experience [4][5][6].

With the continuous growth of the market for and availability of various VR devices, indicators are required for the systematic and rational evaluation ofthe comfort experience of VR HMDs. Such an evaluation is necessary to enhance the user experience, which is essential to the commercialization of VR devices. Thus, the development of evaluation indicators should be prioritized. While research on cybersickness- causing factors has been actively conducted, no study has ranked cybersickness-causing factors for HMDs. Studies that have identified cybersickness- causing factors in HMDs have not provided evaluation indicators for comfort experience assessment.

This study aimed to develop evaluation indicators for the comfort experience of HMDs. A Delphi survey was conducted with cybersickness experts; the survey was based on cybersickness-causing factors identified from previous studies. Furthermore, the relative importance of the evaluation indicators was determined by calculating weights for all dimensions and sub-factors. The results of this study can be used as basic data to evaluate the comfort experience of VR HMDs in future studies.

2. Litrature Review

The literature on experiencing motion sickness indicates that visual perception and motion are the primary factors determining its occurrence. In preceding studies related to experiencing cybersickness in virtual environments (the experience addressed in this study), inconsistencies in visual perception and motion have been commonly identified as causes of general cybersickness. Therefore, this section describes the main features of sensory mismatch and postural instability—two representative cybersickness theories.

2.1 Sensory Conflict Theory

Sensory conflict theory is a fundamental theory that defines motion sickness. It states that conflicts occur when information recorded by different senses of a person for perceiving the world does not match his or her movements [7][8]. For VR, a conflict exists between the perspectives of viewing content and the vestibular sense responsible for balancing it. Consequently, a mismatch occurs in the information delivered by the two concerned sensory organs.

Sub-concepts of the sensory conflict theory include "input conflict, " "output conflict, " and "expectancy violation." Input conflict occurs during information processing because of a mismatch in the information recorded by the input senses. In contrast, output conflicts result from inconsistencies between processed information as information expressed externally. Finally, expectancy violation refers to the conflict that occurs when information collected based on existing experience and knowledge differs from that expected.

2.2 Postural Instability Theory

Postural instability theory emerged from a criticism of the three sub-concepts of sensory conflict theory. The criticism was that, logically, input conflict, output conflict, and expectancy violation do not differ significantly—the theory is abstract, precluding a complete explanation of motion sickness [9]. it is challenging to explain individual differences in the degree of sensory conflicts experienced by individuals for a given stimulation situation.

In contrast to sensoryconflict theory, which postulates mismatches in sensory information as the cause of motion sickness, postural instability theory specifies the cause as a reduction in postural control ability—defined as persistent postural imbalance. Individuals may experience motion sickness when they lose the ability to control their posture under a closed-loop feedback system consisting of "environmental dynamics, " "goals of behavior, " and "constrain the control of posture, " from which they perceive the world [10].

2.3 Cybersickenss Studies on Hardware

Based on previous studies, cybersickness-causing factors can be divided into three categories: content factors, hardware factors, and user characteristic factors [11]. This study focused on hardware (HMD)-related cybersickness- causing factors.

Previous studies regarding hardware-related cybersickness- causing factors have verified the influence of display device types, including screens, monitors, and HMDs, on cybersickness [12][13] [14]. Studies on the changes in the degree of cybersickness by field of view (FoV) [15][16], the influence of time delay on cybersickness [17], the influence of framerate on cybersickness [18], and the influence of flickering on cybersickness have been conducted [19]. Because this study aims to develop evaluation indicators for the comfort experience of VR HMD, we comprehensively included the variables covered in various studies. Each variable is described in detail in the next section.

3. Research Framework

This study classified the evaluation indicators of the comfort experience of HMDs into six dimensions based on the prior studies on cybersickness, focusing on the sensory conflict and postural instability theories. From these dimensions, the critical sub-factors were identified and compared between standards for determining guidelines for the production of HMDs characterized by reduced cybersickness. The standards considered were IEEE standards, which are international standards, TTA, which are South Korea’s domestic standards, and factors presented in previous papers on cybersickness [20][21][22]. The dimensions of the evaluation indicators were FoV, latency, framerate, V-sync, rig, and resolution. The research framework is summarized in Figure 1.

(Figure 1) System framework

3.1 Head-Mounted Display

VR is defined as a computer-system-based convergence technology that provides users with a realistic and immersive sense of reality in an artificially-formed virtual space [23]. HMDs, which users wear on their heads for VR space experience, include high-resolution displays, GPS receivers, earth magnetic fields, and gyroscopes. The first HMD was developed in 1968 by the American computer scientist Ivan Edward Sutherland, who worked at Harvard University, and his student Bob Sproull [24][25]. Since thelaunch of Oculus Rift, HMDs have been actively produced by manufacturers (Table 1) such as Samsung and Oculus (Gear VR, jointly produced by the two companies), HTC (Vive), and others.

(Table 1) HMDs variety of specifications

HMDs in the market have several specifications that have been designed to provide users with an interface that offers a comfortable experience. HMDs artificially create VR and facilitate deep, three-dimensional perception through binocular parallax. This operation principle causes visual and behavioral conflicts, resulting in users experiencing cybersickness, as confirmed by many previous studies [26][27].

3.2 Field of View

FoV is the size of the area that users can observe. It is classified by the vertical/horizontal orientation of the display, viewing angle, and diagonal length of the FoV. If a display is used, a small FoV indicates a narrow viewing area, in which users must move the screen frequently. While a small FoV is characterized by reduced image immersion and visual cognitive ability, a high FoV can cause screen distortion, resulting in users feeling dizzy or uncomfortable. Furthermore, a large device weight because of a high FoV is likely to make users feel less comfortable and more fatigued [28].

Figure 2 illustrates the FoV of the human eye. The human eye vision-level differs for monocular vision, binocular vision, horizontal-line-of-sight view, and vertical-line-of-sight view. For the human eye, for a horizontal-line-of-sight view, monocular and binocular vision cover an FoV of approximately 160º and 120º, respectively. Similarly, the vertical-line-of-sight view has different FoVs for the two types of vision. The concentration of a person depends on the FoV of the human eye. Therefore, the FoV of VR content depends on the purpose of the content, such as therapy, training, or entertainment.

(Figure 2) FoV of human eye [29]

3.3 Latency

Latency refers to the difference between the time required for the VR device to respond to a behavior signal inputted by a user and the time at which the VR device presents the signal. VR latency can distract users and affect their comfort level and the intensity of cybersickness [28]. Studies have demonstrated that users experience dysentery and discomfort when the latency exceeds a certain level (60 ms) and a reduced sense of immersion and fatigue because of losing their sense of direction [30].

Figure 3 illustrates the systematic process by which latency occurs. VR latency is the sum of the times required for device location detection through motion tracking systems, rendering through game engines, scene transmission through graphics hardware, and transmission of pixel-based photons by the display [31].

(Figure 3) Motion to photon latency [32]

3.4 Framerate

Framerate refers to the rate of speed at which a stationary image is reproduced. A video consists of a collection of still pictures, each of which is a frame. The rate at which the frame is changed is the framerate. Figure 4 illustrates the dependence of the image processing frequency on the framerate. Low framerate can cause flickering. The flickering phenomenon can cause eye fatigue and is likely to cause headaches, fatigue, and photo-epileptic seizures. The cumulative effects of the flickering phenomenon include discomfort and cybersickness [28].

(Figure 4) Dependence of the image processing frequency on the framerate [33]

3.5 V-sync

V-sync refers to the number of images that a display projects per second. The higher the V-sync, the more scenes can be displayed in a second, and the lower the number of screen breaks. Typical digital images require an average V-sync of 60 Hz, whereas game content requires a relatively higher V-sync. If the device’s V-sync does not match the frequency of the content’s image, a tear occurs on the screen. The tearing phenomenon caused by a low V-sync affects the device user’s comfort level. V-sync is also a significant factor affecting users’ comfort experience.

Figure 5 illustrates the screen for different V-sync levels. The left monitor’s screen presents a horizontal tear caused by a low V-sync, while the right monitor’s screen displays a clean scene without any tear because the V-sync is adequate.

(Figure 5) Screen of different V-sync [34]

3.6 Rig

A rig is a device used to combine two or more cameras for filming. Several cameras are required for producing images, such as three-dimensional and VR images. There are different types of rigs, such as horizontal, vertical, and 360ºrigs (Figure 6), and the type used depends on the type of image required.

(Figure 6) Different types of rigs [35]

If a rig is used and if the rig’s cameras are not adjusted precisely or aligned close to the no-parallax point, HMD users could feel uncomfortable because of distortion between images. Therefore, it is essential to position the rig precisely when creating content. Because the size of cameras used for producing VR images is large due to their high performance, highly precise adjustments are required for these cameras; their size makes it difficult to align them close to the no-parallax point between optical instruments [28].

3.7 Resolution

The conceptual definition of resolution is the number of pixels on the screen. As illustrated in Figure 7, the number of pixels and display space vary with the resolution. ahigher resolution corresponds to a larger number of pixels on the screen, a larger space, and greater detail.

(Figure 7) Various resolutions

The sense of immersion and the comfort level of users of VR content depend on the resolution. Furthermore, in contrast to ordinary video devices, VR device optical systems involve convex lenses. Therefore, for a comfortable experience, VR device users require a higher resolution than universal image display environments [28].

4. Method

No previous study has ranked the major factors responsible for cybersickness with HMDs. In this study, six dimensions were identified and compared for different standards for determining guidelines for the production of HMDs characterized by reduced cybersickness. The standards considered were IEEE standards, which are international standards, TTA standards, which are South Korea’s domestic standards, and factors obtained from previous studies on cybersickness. The dimensions were FoV, latency, framerate, V-sync, rig, and resolution.

This study used the Delphi method, widely used in technology, education, and policy-making [36]. Because the Delphi method has a score-ranking weakness in which participants cannot identify differences between items with different ranks [37], the analytic hierarchical process (AHP) method was also used. In the AHP method, the entire decision-making process is divided into several stages. Each stage is then analyzed and interpreted stepwise for reasonable decision-making [38]. In this study, the AHP method was also used to identify and rank VR cybersickness-causing factors.

A total of 20 VR experts participated in this study. They comprised officials from a government-funded research institute, VR/AR content development companies, university professors, doctors, and media professionals actively engaged in research on cybersickness reduction. All participants had experienced and studied cybersickness problems directly or indirectly. These experts were chosen based on their knowledge of the cybersickness problem, nature of work, and experience.

This study obtained optimal results by considering the scope of HMD users’ experience. Accordingly, the number of VR/augmented reality contents used and the time taken to view each of the contents were considered, as depicted in Table 2. Furthermore, the type of content preferred by the participating experts was also considered to minimize the experts’ personal bias in evaluating cybersickness-casing factors.

(Table 2) Demographic characteristics of experts participated in Delphi survey

In this study, the measurement items were incorporated into email-based questionnaires to analyze and control data collected from the experts and increase survey result accuracy [39].

4.1 Delphi Survey

The Delphi research method, which involves collecting survey information from experts and collecting opinions, was introduced by Dalkey and Helmer (1963) [40]. It is a tool for experts to organize and prioritize collected factors for solving problems from a different perspective [41], and it has a logical basis. It is based on the decision-making principle that many opinions are more reliable than several. The Delphi survey is conducted mainly through the mail rather than physically from a gathering of experts. Consequently, opinions are provided and collected through email.

The Delphi research method involves repeated surveys from anonymous experts to evolve a collective consensus, without the experts gathering in person to debate. Usually, opinion coordinators in charge of the surveys obtain the opinions of 10–15 experts two or three times. The experts receive feedback from other experts, which produces the average or median value of each survey.

In this study, Delphi expert surveys were conducted four times to develop indicators for evaluating the comfort experience of HMDs. The surveys were conducted between October 1, 2018, and November 2, 2018. In the surveys, concept definitions, types of factors, and indicator pools obtained from a literature review were presented as examples to indicate research direction.

The Delphi research method was conducted using the process presented in Figure 8 and Table 3. Experts assessed the validity based on other experts’opinions collected through open-ended questions. Each expert answered a questionnaire with a seven-point scale. After a statistical analysis of the response results, box and whisker charts were prepared to identify the measurement variable, the average of the experts’ responses, the scores of the experts’ responses, and the quartiles of the experts’ responses. In this process, experts verified the feedback and statistical analysis results prepared by other experts and corrected their presented scores.

(Figure 8) Delphi survey outline

(Table 3) Delphi survey outline for developing

In the first Delphi survey, a validity analysis of an HMD’s dimension classification for cybersickness was conducted, and the definition of each factor was collected. In the second Delphi survey, the dimension classification was re-evaluated, factors were classified, and a validity analysis was conducted based on the dimension classification for HMD cybersickness reduction. In the third Delphi survey, factors based on the dimension classification were re-evaluated, the evaluation indicators of the classified factors were chosen, and a validity analysis was conducted. In the fourth Delphi survey, the factors were re-evaluated, the weights of the evaluation indicators were calculated, and a ranking survey was conducted.

The results of the Delphi survey were analyzed using SPSS 25 (Windows version). Statistics such as mean, percentage, standard deviation, and frequency were computed.

5. Results

5.1 Dimension classification and definition

The factors causing cybersickness in a VR environment were collected and compared against different standards for two purposes: 1) to determine guidelines for producing HMDs characterized by reduced cybersickness and 2) to classify and define the dimensions of the evaluation indicators for the comfort experience of HMDs. The standards considered were IEEE standards, which are international standards, TTA standards, which are South Korea’s domestic standards, and factors obtained from previous papers on cybersickness. In the Delphi surveys, the evaluation frame was presented in six dimensions: FoV, latency, framerate, V-sync, rig, and resolution. As depicted in Figure 9, the evaluation indicators were developed (Evaluating for Comfort of HMD; ECH). Table 4 presents the definitions of the factors.

(Figure 9) Evaluating for Comfort of HMD (ECH)

(Table 4) Definition of factors

Based on the Delphi surveys, the six dimensions of the evaluation indicators were defined. The detailed factors for the dimensions were identified as FoV–device FoV, latency –device latency, framerate–device framerate, V-sync– device V-sync, rig–camera-angle view, rig–no-parallax point, resolution–device resolution, and resolution–pixels per inch (PPI).

5.2 Factor Classification

In the HMD comfort experience evaluation, factor classification was the most fundamental component. It was presented in the Delphi surveys by assigning sub-factors for each dimension based on the literature review; the classification is depicted in Table 5.

(Table 5) Factors classification reference

5.3 Delphi Analysis Result

In the first Delphi survey, the dimensions of HMD-induced cybersickness were classified and defined. Moreover, a validity analysis was conducted based on opinions collected through open-ended questions. A total of 20 experts (100%) participated in the first Delphi survey.

The validity analysis results of the first Delphi survey for the dimension classification and definitions are depicted in Table 6. The experts agreed on the definitions of the evaluation indicator dimensions for the comfort experience evaluation for HMDs, with most central values being 6 (reasonable) and the minimum being 5 (slightly reasonable).

(Table 6) Dimension classification validity analysis

In the second Delphi survey, the HMD cybersickness dimension classification provided in the first Delphi questionnaire was reassessed. A validity analysis of the factor classification based on the dimension classification was performed. In this survey, 15 experts (75%) participated in the re-evaluation through the second Delphi questionnaire, while all 20 experts (100%) participated in the factor classification.

The experts agreed on the results of the revaluation of the HMD cybersickness dimension classification and definitions in the second Delphi survey, with the median value of most dimensions being 6 (reasonable), as depicted in Table 7. The value of "rig"was also relatively higher compared to the first survey. The second step of the survey confirmed the validity of HMD dimension classification and definition.

(Table 7) Re-evaluation of the dimension classification

The validity analysis results of the factor classification based on the dimension classification for HMD-induced cybersickness conducted in the second Delphi survey are presented in Table 8. The experts expressed their opinions on the factor classification for the evaluation indicators for the comfort experience of HMDs. The scored values were mostly 6 (reasonable), and the minimum value was 5 (slightly reasonable).

(Table 8) Factor classification validity analysis

In the third Delphi survey, the HMD cybersickness factor classification obtained in the second Delphi survey was reassessed. The reassessment was performed through the third Delphi questionnaire and involved 14 experts (70%).

The revaluation results in the third Delphi survey were mostly 6 (reasonable), and the minimum value was 5 (slightly reasonable), as depicted in Table 9. The re-evaluation analyses performed in the second step demonstrated the validity of the HMD cybersickness factor classification.

(Table 9) Re-evaluation of factor classification [3rd] Re-evaluation

5.4 Selection of Measurement Indicators

In the third Delphi survey, the indicator pool, which could be quantified and measured, was obtained through a literature review and used to select evaluation indicators and analyze validity. The validity of each factor’s evaluation criteria was assessed on a seven-point scale by referring to the indicator pool. The measurement criteria of the pool presented for each factor are depicted in Table 10.

(Table 10) Measurement criteria of the pool

All 20 experts (100%) participated in the selection and validity analysis of the evaluation indicators in the third Delphi survey.

In this survey, in the validity analysis performed to choose the measurement indicators, the median value for most of the measurement indicators presented was 6 (suitable), as depicted in Table 11.

(Table 11) Selection of measurement indicators delphi result

In the validity analysis of the evaluation indicators for "camera angle view, " a sub-factor of the rig, the median value was 5(slightly reasonable), which revealed relatively lower validity than the other factors. However, this value validates the measurement criteria indicator.

5.5 Calculation of Weights

Based on the indicators chosen through the survey analysis, the weights for each factor were calculated in the fourth Delphi survey, in which all 20 experts (100%) participated.

From the survey results, the evaluation indicators for the comfort experience of VR headsets were developed as follows.

The AHP analysis revealed a consistency index (C.I.) of 0.01, indicating that the weights were consistent by experts’responses. As depicted in Table 12, the final weights for the dimensions were as follows: latency, 0.24; framerate, 0.20; resolution, 0.20; FoV, 0.15; V-sync, 0.11; and rig, 0.11. Therefore, the ranking of the dimensions was as follows: first priority, latency; joint second priority, framerate and resolution; third priority, FoV; and joint fourth priority, V-sync and rig.

(Table 12) ECH ranks and weights

The weights for the factors of the dimensions were as follows: device latency, 0.19; device framerate, 0.16; device resolution, 0.14; PPI, 0.14; device FoV, 0.10; camera angle view, 0.09; no-parallax point, 0.09; and device V-sync, 0.08. Therefore, the ranking of the factors was as follows: first priority, device latency; second priority, device framerate; joint third priority, device resolution and PPI; fourth priority, device FoV; joint fifth priority, camera angle view and no-parallax point; and sixth priority, device V-sync.

6. Conclusion

This study aimedto develop indicators for the practical evaluation of the comfort experience of HMDs by classifying and prioritizing cybersickness-causing factors presented by previous studies and in standard documents. The priorities of the dimensions and factors were derived from the results of the four Delphi expert surveys.

Based on the priorities, latency is the most crucial consideration for the comfort experience of HMDs . the time interval for the user’s real-time operation to be displayed as a video signal in the HMD is the dimension that most affects the user’s comfort level. Next in the priority order are framerate and resolution, implying that device specification that determines the realistic expression of VR through displays strongly influences the user’s sense of immersion and the occurrence of cybersickness. The third priority is FoV, which indicates that HMD users are likely to experience cybersickness if they perceive the viewing angle differently from that in the real world. The lowest priorities are V-sync and the rig, suggesting that, if distortion occurs in the VR image of the HMD, its effect on the comfort experience is relatively smaller than the effects of the preceding dimensions. However, distortion induces cybersickness.

Despite many previous studies regarding VR cybersickness-causing factors, no study has ranked cybersickness- causing factors for HMDs. In this study, the relative importance of the evaluation indicators was determined by calculating weights for all dimensions and sub-factors. The results of this study can be used as a reference for future studies. Moreover, as a practical contribution, developers can conduct technical development while considering each factor’s ranking.

However, considering that this study is a first attempt to rank cybersickness-causing factors for HMDs, additional studies are required to objectify the VR HMDs evaluation index presented in this study. Furthermore, determining how to reflect individual differences due to personal experiences or content- related cybersickness-causing factors should be considered in future research.