Content for TR 22.847 Word version: 18.2.0
5 Use case
5.1 Immersive multi-modal Virtual Reality (VR) application
5.1.1 Description
5.1.2 Pre-conditions
5.1.3 Service Flows
5.1.4 Post-conditions
5.1.5 Existing features partly or fully covering the use case functionality
5.1.6 Potential New Requirements needed to support the use case
...
...
5 Use case p. 10
5.1 Immersive multi-modal Virtual Reality (VR) application p. 10
5.1.1 Description p. 10
Immersive multi-modal VR application describes the case of a human interacting with virtual entities in a remote environment such that the perception of interaction with a real physical world is achieved. Users are supposed to perceive multiple senses (vision, sound, touch) for full immersion in the virtual environment. The degree of immersion achieved indicates how real the created virtual environment is. Even a tiny error in the preparation of the remote environment might be noticed, as humans are quite sensitive when using immersive multi-modal VR applications. Therefore, a high-field virtual environment (high-resolution images and 3-D stereo audio) is essential to achieve an ultimately immersive experience.
One of the major objectives of VR designers and researchers is to obtain more realistic and compelling virtual environments. As the asynchrony between different modalities increases, users' sense of presence and realism will decrease. There have been efforts (since 1960s or even earlier) in multi-modal-interaction research regarding the detection of synchronisation thresholds. The obtained results vary, depending on the kind of stimuli and the psychometric methods employed. Hirsh and Sherrick measured the synchronisation thresholds regarding visual, auditory and tactile modalities [23].
Figure 5.1.1-1: The synchronisation thresholds regarding visual, auditory and tactile modalities measured by Hirsh and Sherrick
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
M.E. Altinsoy and co. believe the audio-tactile synchronization has to be at least within an accuracy of ±40 ms [22]. More results have been reported based on extensive theoretical and experimental efforts. [24] further indicated the perceptual threshold values were 50 ms for the audio lag and the 25 ms for audio lead.
As to the visual-tactile synchronisation threshold, Massimiliano Di Luca and Arash Mahnan provided test results in [25] that indicate that none of the participants could reliably detect the asynchrony if haptic feedback was presented less than 50ms after the view of the contact with an object. The asynchrony tolerated for haptic before visual feedback was instead only 15ms.
5.1.2 Pre-conditions p. 11
The devices for immersive multi-modal VR application may include multiple types of devices such as VR glass type device, the gloves and other potential devices that support haptic and/or kinaesthetic modal. These devices which are 5G UEs are connected to the immersive multi-modal VR application server via the 5G network without any UE relays, see Figure 5.1.2-1.
Based on the service agreement between MNO and immersive multi-modal VR application operator, the application operator may in advance provide the 5G network with the application information including the application traffic characteristics and the service requirement for network connection. For example, the packet size for haptic data is related to the Degrees of Freedom (DoF) that the haptic devices supports, and packet size for one DoF is 2-8 Bytes [3] and the haptic device generates and sends 500 haptic packets within one second.
Figure 5.1.2-1: Immersive multi-modal VR application with multiple 5G UEs directly connected to 5G network
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
5.1.3 Service Flows p. 11
1)
The application user utilizes the devices to experience immersive multi-modal VR application. The user powers on the devices to connect to the application server, then the user starts the gaming application.
2)
During the gaming running period, the devices periodically send the sensing information to the application server, including: haptic and/or kinesthetic feedback signal information which is generated by haptic device, and the sensing information such as positioning and view information which is generated by the VR glasses.
3)
According to the uplink data from the devices, the application server performs necessary process operations on immersive game reality including rendering and coding the video, the audio and haptic model data, then application server periodically sends the downlink data to the devices, with different time periods respectively, via 5G network.
4)
The devices, respectively, receive the data from the application server and present the related sensing including video, audio and haptic to the user.
5.1.4 Post-conditions p. 12
The user experiences the immersive game reality application enabled by 5G network, and the 5G system address the service requirements of the application.
5.1.5 Existing features partly or fully covering the use case functionality p. 12
3GPP TS 22.261 specifies KPIs for high data rate and low latency interactive services including Cloud/Edge/Split Rendering, Gaming or Interactive Data Exchanging, Consumption of VR content via tethered VR headset, and audio-video synchronization thresholds.
Support of audio-video synchronisation thresholds has been captured in TS 22.261:
Due to the separate handling of the audio and video component, the 5G system will have to cater for the VR audio-video synchronisation in order to avoid having a negative impact on the user experience (i.e. viewers detecting lack of synchronization). To support VR environments, the 5G system shall support audio-video synchronisation thresholds:
- in the range of [125 ms to 5 ms] for audio delayed and
- in the range of [45 ms to 5 ms] for audio advanced.
5.1.6 Potential New Requirements needed to support the use case p. 12
[PR 5.1.6-1]
The 5G System shall provide the network connection to address the KPIs for immersive multi-modal VR applications, see Table 5.1.6-1.
| Use Cases | Characteristic parameter (KPI) | Influence quantity | Remarks | |||||
|---|---|---|---|---|---|---|---|---|
| Max allowed end-to-end latency | Service bit rate: user-experienced data rate | Reliability | Message size (byte) | # of UEs | UE Speed | Service Area | ||
| Immersive multi-modal VR (UL: device → application server) | 5 ms
(note 2) | 16 kbit/s -2 Mbit/s
(without haptic compression encoding); 0.8 - 200 kbit/s (with haptic compression encoding) | [99.9%]
(without haptic compression encoding) [99.999%] (with haptic compression encoding) | 1 DoF: 2-8 3 DoFs: 6-24 6 DoFs: 12-48 More DoFs can be supported by the haptic device | - | Stationary or Pedestrian | typically
< 100 km² (note 3) | Haptic feedback |
| 5 ms | < 1Mbit/s | [99.99%] | MTU | - | Stationary or Pedestrian | typically
< 100 km² (note 3) | Sensor information e.g. position and view information generated by the VR glasses | |
| Immersive multi-modal VR (DL: application server → device) | 10 ms
(note1) | 1-100 Mbit/s | [99.9%] | 1500 | - | Stationary or Pedestrian | typically
< 100 km² (note 3) | Video |
| 10 ms | 5-512 kbit/s | [99.9%] | 50 | - | Stationary or Pedestrian | typically
< 100 km² (note 3) | Audio | |
| 5 ms
(note 2) | 16 kbit/s -2 Mbit/s
(without haptic compression encoding); 0.8 - 200 kbit/s (with haptic compression encoding) | [99.9%]
(without haptic compression encoding) [99.999%] (with haptic compression encoding) | 1 DoF: 2-8 3 DoFs: 6-24 6 DoFs: 12-48 | - | Stationary or Pedestrian | typically
< 100 km² (note 3) | Haptic feedback | |
|
NOTE 1:
Motion-to-photon delay (the time difference between the user's motion and corresponding change of the video image on display) is less than 20 ms, the communication latency for transferring the packets of one audio-visual media is less than 10 ms, e.g. the packets corresponding to one video/audio frame are transferred to the devices within 10 ms.
NOTE 2:
According to IEEE 1918.1 [3] as for haptic feedback, the latency is less than 25 ms for accurately completing haptic operations. As rendering and hardware introduce some delay, the communication delay for haptic modality can be reasonably less than 5 ms, i.e. the packets related to one haptic feedback are transferred to the devices within 10 ms.
NOTE 3:
In practice, the service area depends on the actual deployment. In some cases a local approach (e.g. the application servers are hosted at the network edge) is preferred in order to satisfy the requirements of low latency and high reliability.
|
||||||||
[PR 5.1.6-2]
Due to the separate handling of the multiple media components, synchronization between different media components is critical in order to avoid having a negative impact on the user experience (i.e. viewers detecting lack of synchronization). Applying synchronization thresholds in the 5G system may be helpful in support of immersive multi-modal VR applications when the synchronization threshold between two or more modalities is less than the latency KPI for the application. Typical synchronization thresholds (see [22], [23], [24] and [25]) are summarised in Table 5.1.6-2.
| synchronization threshold (note 1) | ||
|---|---|---|
| audio-tactile | audio delay:
50 ms | tactile delay:
25 ms |
| visual-tactile | visual delay:
15 ms | tactile delay:
50 ms |
|
NOTE 1:
For each media component, "delay" refers to the case where that media component is delayed compared to the other
|
||
[PR 5.1.6-3]
The 5G network shall support a mechanism to allow an authorized 3rd party to provide QoS policy for multiple flows (e.g., haptic, audio and video) of multiple UEs associated with a multi-modal application. The policy may contain e.g. coordination information.
[PR 5.1.6-4]
The 5G system shall support a mechanism to apply 3rd party provided policy for flows associated with an application. The policy may contain e.g. coordination information.
