KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Overview of Motion-to-Photon Latency Reduction for Mitigating VR Sickness

  • Ryu, Yeongil (Department of Computer Education, Sungkyunkwan University) ;
  • Ryu, Eun-Seok (Department of Computer Education, Sungkyunkwan University)
  • Received : 2021.01.06
  • Accepted : 2021.06.10
  • Published : 2021.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

For several years, virtual reality (VR) and augmented reality (AR) technologies have been improving. However, some hurdles remain that slow down the distribution of VR and AR devices, such as head-mounted display (HMD), and related consumer content. One issue is VR motion sickness, which has been experienced by users using 360 degree VR content via HMD. This paper discusses the related international standardization work that classifies the factors causing VR sickness, and proposes the process for VR sickness level evaluation. Among the factors causing VR sickness, many research institutes regard minimizing MTP (Motion-to-Photon) latency as the key enabler to mitigate VR sickness. Thus, this paper introduces research trends of MTP latency measurement and MTP latency mitigation. This paper categorizes the research on MTP latency measurement into 2 categories of hardware-based approach and software code-level approach. The 2 approaches have different pros and cons depending on use-case, purpose, and architecture of each multimedia system. The pros and cons are addressed in this paper. Additionally, the research on mitigating MTP latency with diverse strategies such as proactive computing, caching, and edge server technology is explained, and compared to conventional technologies, shows improved performance.

Keywords

1. Introduction

Recent developments in the 360 degree video technology for virtual reality (VR) and augmented reality (AR) applications are key next-generation factors in the field of multimedia technology. Therefore, the Moving Picture Experts Group (MPEG), an international standard organization, launched the MPEG-I (Coded Representation of Immersive Media, ISO/IEC 23090) project in 2016 to develop standards for immersive media technologies, i.e., 3 degrees of freedom (3DoF), 3DoF+, 6DoF, and light fields. The project is expected to be completed by 2021 [1,2].

Conversely, several factors have been slowing down the distribution of VR and AR devices, such as head-mounted display (HMD), and related consumer content. One issue is VR motion sickness, which has been experienced by users using 360 degree VR content via HMD [3]. Users experience motion sickness while using 360 degree VR content, owing to the disharmony between human motion and the senses from sensory organs in skin and muscles, eyes, and vestibular organs (Fig. 1).

E1KOBZ_2021_v15n7_2531_f0001.png 이미지

Fig. 1. Three sensory organs, and the cause of motion sickness.

IEEE, another international standard organization, launched the IEEE 3079 standard (IEEE standard for HMD-based VR sickness reduction technology) [4]. Unlike MPEG-I, IEEE 3079 standard focuses on the reduction of VR motion sickness. The IEEE 3079 standard defines the factors that affect the user’s VR sickness, and suggests the process for analyzing and evaluating the user’s VR sickness level in a quantitative domain. The standardization project for the IEEE 3079 was started in 2017, and it is expected to be finalized by 2021. Fig. 2 shows the timeline of the standardization of IEEE 3079 and related standard.

E1KOBZ_2021_v15n7_2531_f0002.png 이미지

Fig. 2. The IEEE 3079 standardization project.

The factors that affect user VR sickness include the continuity of viewpoint, 3D sound, stitching optimization, latency minimization, resolution and flicker optimization, as shown in Table 1. IEEE 3079 surveyed the correlation between the factors and user motion sickness. IEEE 3079 used the result of the survey for subjective motion sickness evaluation and user bio-information for objective motion sickness evaluation. Based on the subjective and objective evaluation tools, IEEE 3079 proposed the VR sickness evaluation process and measurement metric in a quantitative domain. Fig. 3 depicts the VR sickness evaluation process of IEEE 3079, which can be useful for the VR sickness level of diverse VR multimedia systems with HMD and can be an indicator inducing VR sickness mitigation.

Table 1. Factors causing VR sickness.

E1KOBZ_2021_v15n7_2531_t0001.png 이미지

E1KOBZ_2021_v15n7_2531_f0003.png 이미지

Fig. 3. VR sickness level evaluation process in IEEE 3079.

Among the factors distinguished by IEEE 3079, Motion-to-photon (MTP) latency significantly induces the disharmony of the aforementioned senses when users experience 360 degree VR content via HMD. MTP refers to the delay between the motion of the user, and the corresponding video shown on the display [5]. A lengthy MTP latency time intensifies VR motion sickness. Fig. 4 shows that MTP latency on HMD-based VR systems is caused by delays in motion sensor input and computation, motion sensor data communication, application computation, rendering computation, and display refresh [6]. Depending on the use of various applications, delays in application computation may refer to delays in video data decoding computation and communication. In addition, delays in the display refresh depend on the refresh rate of the HMD display. Fig. 5 shows that regular displays with refresh rates of (60 and 30) Hz have average delays of (8.3 and 16.6) ms, respectively; in the worst-case scenario, the refresh rates for (60 and 30) Hz can be as bad as (16.6 and 33.3) ms, respectively.

E1KOBZ_2021_v15n7_2531_f0004.png 이미지

Fig. 4. MTP latency in HMD systems and its causes.

E1KOBZ_2021_v15n7_2531_f0005.png 이미지

Fig. 5. Output delays depending on the display refresh rate.

As explained above, MTP latency has different characteristics depending on the multimedia system. Researchers have developed techniques to measure and reduce MTP latency, and such technologies are very important for the expansion of the VR and AR markets.

The remainder of the paper is structured as follows. Sections 2 and 3 discuss research trends on techniques and technologies to measure and reduce MTP latency, while Section 4 concludes the paper.

2. Trends of Research on MTP Latency Measurement

Several studies have considered different techniques to measure MTP latency, depending on the use and features of the target multimedia system. The studies can be classified into two categories: (1) studies that design and implement a hardware-based MTP latency measurement system, and (2) studies that suggest an MTP latency measurement technique based on a software code level. Sections 2.1 and 2.2 review research for the aforementioned categories, and discuss the limitations of the studies.

2.1 Time Sequential MTP Latency Measurement System for HMD Developed by

Sogang University

In 2018, Choi et al. at Sogang University developed a hardware-based MTP latency measurement system using a device that mechanically mimics the motions of the human head model [7]. The proposed system comprises a motion generator that creates motions similar to human head movements and sightline changes based on the neck motion data of the user [8], and a photodetector system that measures the final delay of the HMD screen using photodiode modules.

The motion generator uses servo motors to implement yaw and pitch rotations of the neck of the user. However, to simplify the system, the module does not consider roll rotation, which is less frequent. Fig. 6 shows that the motion generator mimics the joints and links in the human head using servo motors; the system uses forward kinematics [9,10] to generate motions based on predefined user movement scenarios.

E1KOBZ_2021_v15n7_2531_f0006.png 이미지

Fig. 6. Kinematic modeling of the motion generator.

The photodetector system comprises photodiode modules and a display. A photodiode is a semiconductor diode that generates output voltages proportional to the brightness of the input light (Fig. 7) [11]. The system uses such attributes of photodiodes to calculate the output voltage corresponding to the brightness of the input light. The output voltage can be reverse-converted to calculate brightness as well.

E1KOBZ_2021_v15n7_2531_f0007.png 이미지

Fig. 7. Mechanism of a photodiode.

The proposed system measures the MTP latency through the following processes: First, it configures a motion sequence that corresponds to a pre-defined scenario. Next, the sequence is entered into the motion generator and photodetector system. The motion generator creates physical movements based on the input motion sequence, and measures the resulting physical rotation angles. The output of the photodetector system is four square objects placed on a playing image.

The brightness level of the square objects is determined by finding the corresponding motion sequence according to the mapping table listing the pre-defined brightness–rotation angle. Fig. 8 shows that the brightness of the two right-hand-side objects represents the yaw angle, while the brightness of the two left-hand-side objects represents the pitch angle. When these four square objects are overlaid on the screen, the photodiode modules receive the brightness information of the output objects to generate voltages that correspond to the brightness levels. Through this process, the photodetector system obtains the current rotation angle information output on the display.

E1KOBZ_2021_v15n7_2531_f0008.png 이미지

Fig. 8. Square objects that are placed on a scene to indicate yaw and pitch rotation angles.

The proposed system compares the timing of the physical rotation angles measured by the motion generator, and that of the rotation angles measured by the photodetector system, thereby calculating the MTP latency in real time. Fig. 9 shows the system mechanism.

E1KOBZ_2021_v15n7_2531_f0009.png 이미지

Fig. 9. Block diagram of the MTP measurement system.

The MTP latency measurement system proposed by Choi et al. at Sogang University is different from those proposed in other studies. This system generates physical motions based on pre-defined scenarios and measures MTP latency in real time. However, the study does not consider natural scene content, and focuses on computer-generated content. Another shortcoming is that the proposed system only supports the measurement of 2 DoF (yaw and pitch).

2.2 Software Code-Level MTP Latency Measurement by Fraunhofer HHI

Fraunhofer HHI, the German research institute, suggested a software code-level MTP latency measurement technique while developing a low-latency cloud-based streaming system for volumetric videos in 2020 [12]. The proposed system does not directly transmit volumetric videos from the server to the client [13]. Rather, the server renders the 3D view of the volumetric video that corresponds to the current viewport of the user, after which the rendered 3D view is converted to 2D images that are compressed and transmitted to the client. This method allows the client to offload 3D video processing into the server, thereby reducing the load on the client, and increasing the MTP latency. To reduce the MTP latency, the Fraunhofer HHI considered a software code-level approach, which can yield efficient processing, to measure the MTP latency.

For accurate MTP latency measurement, the study defined MTP latency in the volumetric video streaming system as follows:

\(\mathrm{T}_{\mathrm{m} t \mathrm{p}}=\mathrm{T}_{\text {server }}+\mathrm{T}_{\text {network }}+\mathrm{T}_{\text {client }}\)       (1)

Tserver, Tnetwork, and Tclient are defined in Eqs. (2)-(4), respectively: 

\(\mathrm{T}_{\text {server }}=\mathrm{T}_{\text {rend }}+\mathrm{T}_{\text {enc }}\)       (2)

\(\mathrm{T}_{\text {network }}=\mathrm{T}_{\text {up }}+\mathrm{T}_{\text {down }}+\mathrm{T}_{\text {trans }}\)       (3)

\(\mathrm{T}_{\text {client }}=\mathrm{T}_{\mathrm{dec}}+\mathrm{T}_{\text {disp }}\)       (4)

Equation (2) refers to the delay time occurring on the server side, where Trend refers to the time required to render the volumetric images corresponding to the current viewport of the user; and Tenc refers to the time required to convert the 3D view of the rendered volumetric video to 2D images, and then encode using a video codec, such as H.264/AVC [14] or H.265/HEVC [15].

Equation 3 refers to the total delay time that occurred during the server–client data communication, where Tup and Tdown represent the propagation delay between the server and the client; and Ttrans refers to the time required to transmit the data.

Equation (4) refers to the delay time on the client side, where Tdec is the time required to decode the received video data; and Tdisp refers to the delay time for outputting the decoded picture on the screen.

Fig. 10 shows a schematic of how Eq. (1) is applied to a cloud-based volumetric video streaming system. In the study, the system makes software code-level recordings of time Ri, when the client application requests frame Fi corresponding to the viewport of the user, and time Di, when the requested frame is received and output to the display to measure the MTP latency based on Eq. (1) and Fig. 10. Then, to measure the MTP latency in a simplified manner, T𝑚𝑡𝑝 is obtained from Di−Ri.

E1KOBZ_2021_v15n7_2531_f0010.png 이미지

Fig. 10. Elements of MTP latency in a volumetric video streaming system.

The MTP latency measurement technique of Fraunhofer HHI is different from the technique developed by Choi et al. at Sogang University; the Fraunhofer HHI technique is easier to implement than the hardware-based MTP latency measurement method, and considers delays occurring on the network.

3. Trends of Research on MTP Latency Reduction

Along with the MTP latency measurement, MTP latency reduction is an essential element for the quality of experience (QoE) in immersive 360 degree VR content. Key MTP latency reduction technologies include mmWave communication, mobile edge computing, and proactive caching. Table 2 lists information about the requirements for providing highly reliable VR services, and key technologies necessary to satisfy each requirement [6]. The following sections discuss studies based on edge computing and proactive caching technology that support low-latency computing.

Table 2. Requirements and key technologies for highly reliable VR services.

E1KOBZ_2021_v15n7_2531_t0002.png 이미지

3.1 Adaptive Cropping Technique by AT&T Labs Research

In 2019, AT&T Labs Research in the US proposed a technique that reduces the MTP latency by proactively caching technique and edge servers [16]. The proposed technique reduces the MTP latency during the server–client streaming of 360 degree game videos. Rather than streaming the entire 360 degree view, the system crops only the field of view (FoV) of the user for lower latency and computation. However, when the MTP latency is high, the margins surrounding the FoV are cropped, and additionally transmitted with the video data of FoV. In doing so, the system can output the video data of the additionally transmitted margin areas when the viewport of the user is outside the current FoV, thereby increasing the QoE of the streaming service, even when MTP latency is high, as shown in Fig. 11.

E1KOBZ_2021_v15n7_2531_f0011.png 이미지

Fig. 11. Adaptive cropping technique considering MTP latency.

The size of the additionally transmitted margin areas is determined based on the current MTP latency. It is determined in real time based on the pre-defined table analyzing the MTP latency and required margin sizes. This table deals with MTP latencies of (50, 100, 300, and 500) ms, providing test results that indicate the optimal size of viewpoint that can cover the user’s movement at a specific MTP latency. Then, it maps the size of the margin areas that can fully (100 %) cover the corresponding scope of movements.

Compared to the conventional method of streaming the entire 360 degree view, the proposed method reduced the bandwidth by 80 %. Moreover, this enables 8K live VR streaming.

3.2 MTP Latency Reduction Technique for Tile-Based 360 degree Video Streaming Systems by UT Dallas

In 2018, UT Dallas in the US proposed a caching technique for tile-based 360 degree video streaming systems in edge computing architecture [17]. Fig. 12 shows that tile-based video streaming divides the video into multiple tiles, and streams only necessary tiles [18, 19, 20, 21]. Compared with the conventional streaming of 8K, 16K, and other ultra-high-definition videos, this method can highly reduce the required network bandwidth and computational complexity [22, 23, 24].

E1KOBZ_2021_v15n7_2531_f0012.png 이미지

Fig. 12. Example of tile-based video streaming.

The proposed technique pre-analyzes the movement patterns of a user for each video content, thereby prioritizing each tile based on the frequency with which each tile is included in the FoV. To reduce the MTP latency, the video tiles are cached on the edge server based on priority.

The study analyzed the movement patterns of the user for each content section using the user viewpoint information dataset [25] in 360 degree videos, which was presented by Lo et al. in 2017. Using the analysis results, the proposed system configures a heatmap of each tile in given intervals (Fig. 13). The heatmap represents the possibility that each tile can be included in the FoV. The proposed technique uses this information to prioritize caching tiles with a higher possibility of being included in the FoV, thereby reducing the MTP latency.

E1KOBZ_2021_v15n7_2531_f0013.png 이미지

Fig. 13. Example of user motion pattern heatmap.

The study conducted tests that compare the cache hit rate of the proposed method with that of generally used caching techniques, such as least recently used (LRU), and least frequently used (LFU) [26], caching methods. The test results indicated that compared with LRU and LFU, the proposed method had (20–40) % higher hit rates.

3.3 Saliency Map-based Bit-rate Adaptation for VR streaming by Sungkyunkwan University

Lee et al. at Sungkyunkwan University proposed the motion-constrained tile set (MCTS)-based 360 degree VR streaming system with bit-rate adaptation technique using saliency map in 2019 [27]. The tile-based video streaming method is a highly efficient technique for VR multimedia systems with HMD, because it transmits only the video data in the user’s viewport. Therefore, compared to the conventional approach that transmits the entire pictures, a tile-based approach can achieve higher video quality at limited network bandwidth [28].

Although the tile-based approach has advantages in 360 degree video streaming, the prediction in the process to decode tiled video has a few hurdles. This is because when the reference video data was outside previous viewports at a client side, the reference video data of the tiles in the current viewport can be lost. Therefore, the server side in 360 degree video streaming systems needs to consider the coding structure that restricts the available search range for inter-picture prediction at the encoder level.

To solve this problem in tile-based 360 degree video streaming systems, MCTS was introduced. MCTS technique controls the search range for reference during temporal prediction to be in the viewport of the previous picture. This coding tool enables independent decoding of each tile at the client side, regardless of the movement of the user’s viewport. Using MCTS achieves dramatic Bit-rate reduction in the entire streaming process, although the coding efficiency of each tile is slightly lower than the conventional coding scheme, owing to the restriction of the search range in temporal prediction [29,30]. Son et al. implemented and tested the MCTS with HM (HEVC test model software) and SHM (SHVC test model software) in 2018 [31]. According to Son et al., MCTS in the tile-based video streaming system with 9 tiles achieves a bit-rate reduction of about 48 %, compared to the conventional HM [15] and SHM [32]. However, many factors, such as the number of tiles and the size of viewport, can change this result.

To achieve improved performance, Lee et al. employs saliency detection technique [33, 34, 35], as well as MCTS for viewport-dependent video streaming systems, such as 360 degree video streaming with HMD. The saliency detection in the research finds a probability distribution of user eye fixation information based on a convolutional neural network (CNN). Fig. 14 shows an example of saliency map with the Slient360! dataset [36], which is a platform for evaluating the performance of saliency mapping models. The authors assigned priorities to each tile. For example, tiles that have a higher probability are assigned higher priority and more bits than others. In doing so, the importance of each tile can be known, and more bits can be assigned to more important tiles for encoding. Table 3 outlines a brief procedure for the assigning of priority and bits based on a given network bandwidth and saliency map.

E1KOBZ_2021_v15n7_2531_f0014.png 이미지

Fig. 14. Example of saliency map. (a)-(c) L Test image in the Salient360! Dataset. (d)-(f): Corresponding saliency map prediction from the CNN-based saliency model.

Table 3. Proposed procedure of the quality adaptation based on saliency map.

E1KOBZ_2021_v15n7_2531_t0003.png 이미지

Table 4 shows that at 4 × 4, 6 × 3, 8 × 4, and 12 × 6 tiling, the proposed method achieves a BD-rate reduction of an average (17.2, 17.9, 18.9, and 20) %, respectively. The proposed method focuses on maximizing video quality at a limited network bandwidth, and it can be considered to achieve MTP latency optimization by the proactive computing technique.

Table 4. Bit-rate reduction of the proposed MCTS technique with saliency map

E1KOBZ_2021_v15n7_2531_t0004.png 이미지

4. Conclusion

This paper discussed hurdles for immersive 360 degree videos and VR/AR applications, and the related international standardization work was explained, which classifies the factors causing VR sickness, and proposes the process for VR sickness level evaluation.

Among the factors causing VR sickness, many research institutes regard MTP latency as the key enabler to mitigate VR sickness. Thus, this paper introduced the research trends on MTP latency measurement, and categorized the research on MTP latency measurement into 2 categories of hardware-based approach, and software code-level approach. Furthermore, the pros and cons of each research effort were discussed.

Additionally, this paper introduced the key enablers for the highly reliable VR multimedia system, and research on mitigating MTP latency. The research employs proactive computing, caching, and edge server technology, and compared to conventional technologies, shows improved performance.

Acknowledgement

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2019R1A2C1010476).

References

  1. M. Domanski, O. Stankiewicz, K. Wegner, and T. Grajek, "Immersive visual media - MPEG-I: 360 video, virtual navigation and beyond," in Proc. of 2017 International Conference on Systems, Signals and Image Processing, Poznan, pp. 1-9, 2017.
  2. M. Wien, J. M. Boyce, T. Stockhammer, and W. Peng, "Standardization Status of Immersive Video Coding," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 5-17, 2019. https://doi.org/10.1109/jetcas.2019.2898948
  3. R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre, "Recent advances in augmented reality," IEEE Comput. Graph. Appl., vol. 21, no. 6, pp. 34-47, 2001. https://doi.org/10.1109/38.963459
  4. IEEE Std 3079-2020, IEEE, 2020.
  5. B. Iribe, "Virtual reality-A new frontier in computing," in Proc. of the AMD Announces 2013 Developer Summit, San Jose, CA, USA, 2013.
  6. M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, "Toward Low-Latency and Ultra-Reliable Virtual Reality," IEEE Network, vol. 32, no. 2, pp. 78-84, 2018. https://doi.org/10.1109/MNET.2018.1700268
  7. S. W. Choi, S. Lee, Seo, M. W. Seo, and S. J. Kang, "Time sequential motion-to-photon latency measurement system for virtual reality head-mounted displays," Electronics, vol. 7, no. 9, pp. 171, 2018. https://doi.org/10.3390/electronics7090171
  8. M. W. Seo, S. W. Choi, S. L. Lee, E. Y. Oh, J. S. Baek, and S. J. Kang, "Photosensor-based latency measurement system for head-mounted displays," Sensors, vol. 17, no. 5, pp. 1112, 2017. https://doi.org/10.3390/s17051112
  9. P. M. Sharkey, D. W. Murray, and J. J. Heuring, "On the kinematics of robot heads," IEEE Transactions on Robotics and Automation, vol. 13, no. 3, pp. 437-442, 1997. https://doi.org/10.1109/70.585904
  10. J. Funda, R. H. Taylor, and, R. P. Paul, "On homogeneous transforms, quaternions, and computational efficiency," IEEE Transactions on Robotics and Automation, vol. 6, no. 3, pp. 382-388, 1990. https://doi.org/10.1109/70.56658
  11. SM05PD Mounted Photodiodes - SM05 and SM1 Compatible Specifications.
  12. S. Gul, D. Podborski, T. Buchholz, T. Schierl, and C. Hellge, "Low-latency cloud-based volumetric video streaming using head motion prediction," in Proc. of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 27-33, 2020.
  13. S. Shi, and C.-H. Hsu, "A Survey of Interactive Remote Rendering Systems," ACM Comput. Surv., vol. 47, no. 4, pp. 1-29, 2015.
  14. T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the H.264/AVC video coding standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560-576, 2003. https://doi.org/10.1109/TCSVT.2003.815165
  15. G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, "Overview of the High Efficiency Video Coding (HEVC) Standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649-1668, 2012. https://doi.org/10.1109/TCSVT.2012.2221191
  16. S. Shi, V. Gupta, M. Hwang, and R. Jana, "Mobile VR on edge cloud: a latency-driven design," in in Proc. of the 10th ACM Multimedia Systems Conference, pp. 222-231, 2019.
  17. A. Mahzari, A. Taghavi Nasrabadi, A. Samiei, and R. Prakash, "FoV-aware edge caching for adaptive 360 video streaming," in Proc. of the 26th ACM international conference on Multimedia, pp. 173-181, 2018.
  18. Y. Bao, H. Wu, T. Zhang, A. A. Ramli, and X. Liu, "Shooting a moving target: Motion-prediction-based transmission for 360-degree videos," in Proc. of IEEE International Conference on Big Data, pp. 1161-1170, 2016.
  19. J. He, M. A. Qureshi, L. Qiu, J. Li, F. Li, and L. Han, "Rubiks: Practical 360-Degree Streaming for Smartphones," in Proc. of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pp. 482-494, 2018.
  20. D. Ochi, Y. Kunita, A. Kameda, A. Kojima, and S. Iwaki, "Live streaming system for omnidirectional video," in Proc. of IEEE Virtual Reality, pp. 349-350, 2015.
  21. F. Qian, B. Han, L. Ji, and V. Gopalakrishnan, "Optimizing 360 video delivery over cellular networks," in Proc. of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, pp. 1-6, 2016.
  22. M. Graf, C. Timmerer, and C. Mueller, "Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation," in Proc. of the 8th ACM on Multimedia Systems Conference, pp. 261-271, 2017.
  23. M. Hosseini and V. Swaminathan, "Adaptive 360 VR Video Streaming: Divide and Conquer," in Proc. of 2016 IEEE International Symposium on Multimedia, San Jose, pp. 107-110, 2016.
  24. P. R. Alface, J. Macq and, N. Verzijp, "Interactive omnidirectional video delivery: A bandwidth-effective approach," Bell Labs Technical Journal, vol. 16, no. 4, pp. 135-147, 2012. https://doi.org/10.1002/bltj.20538
  25. C. Wu, Z. Tan, Z. Wang, and S. Yang, "A dataset for exploring user behaviors in VR spherical video streaming," in Proc. of the 8th ACM on Multimedia Systems Conference, pp. 193-198, 2017.
  26. H. Chen and Y. Xiao, "Cache access and replacement for future wireless Internet," IEEE Communications Magazine, vol. 44, no. 5, pp. 113-123, 2006. https://doi.org/10.1109/MCOM.2006.1637956
  27. S. Lee, D. Jang, J. Jeong, and E. S. Ryu, "Motion-constrained tile set based 360 degree video streaming using saliency map prediction," in Proc. of the 29th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 20-24, 2019.
  28. de la Fuente, Y. Sanchez, R. Skupin, and T. Schierl, "Video processing for panoramic streaming using HEVC and its scalable extensions," Multimedia Tools and Applications, vol. 76, no. 4, pp. 5631-5659, 2017. https://doi.org/10.1007/s11042-016-4097-4
  29. A. Zare, A. Aminlou, M. M. Hannuksela, and M. Gabbouj, "HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications," in Proc. of the 24th ACM international conference on Multimedia, pp. 601-605, 2016.
  30. C. Feldmann, C. Bulla, and B. Cellarius, "Efficient stream-reassembling for video conferencing applications using tiles in HEVC," in Proc. of the International Conferences on Advances in Multimedia, pp. 130-135, 2013.
  31. J. Son, D. Jang, and E. S. Ryu, "Implementing motion-constrained tile and viewport extraction for VR streaming," in Proc. of the 28th ACM SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 61-66, 2018.
  32. J. M. Boyce, Y. Ye, J. Chen, and A. K. Ramasubramonian, "Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 1, pp. 20-34, 2016. https://doi.org/10.1109/TCSVT.2015.2461951
  33. D. D. Salvucci, and J. H. Goldberg, "Identifying fixations and saccades in eye-tracking protocols," in Proc. of the symposium on Eye tracking research & applications, pp. 71-78, 2000.
  34. X. Dong, Y. Yan, M. Tan, Y. Yang, and I. W. Tsang, "Late Fusion via Subspace Search with Consistency Preservation," IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 518-528, 2019. https://doi.org/10.1109/tip.2018.2867747
  35. E. Vig, M. Dorr, and D. Cox, "Large-scale optimization of hierarchical features for saliency prediction in natural images," in Proc. of the IEEE conference on computer vision and pattern recognition, pp. 2798-2805, 2014.
  36. J. Gutierrez, E. J. David, A. Coutrot, M. P. Da Silva, and P. L. Callet, "Introducing UN Salient360! Benchmark: A platform for evaluating visual attention models for 360° contents," in Proc. of 2018 Tenth International Conference on Quality of Multimedia Experience, pp. 1-3, 2018.