Content for  TR 26.998  Word version:  18.1.0

Based on the use cases, the following formats, codecs and packaging formats are of relevance for interactive immersive media distribution of AR:

• Scene graph and scene description
• 2D video formats
• 3D formats such as static and dynamic point clouds or meshes
• 2D video formats with depth
• Audio formats supporting mono, stereo, and/or spatial audio
• Several video decoding instances
• Decoding tools for such formats
• Low-latency downlink real-time streaming of the above media
• Uplink streaming of pose information and interaction data

The above scenarios relate to the following cases in clause 6 of TR 26.928. In particular:

• For STAR:
  • Viewport-dependent streaming based on clause 6.2.3 of TR 26.928,
  • Raster-based split rendering based on clause 6.2.5 of TR 26.928,
  • Generalized XR split rendering based on clause 6.2.6 of TR 26.928.
• For EDGAR:
  • Raster-based split rendering based on clause 6.2.5 of TR 26.928.

For STAR-based delivery, a basic architecture as shown in Figure 6.3.6-1 applies. Two important latency considerations are important:

• User interaction latency, i.e. the time duration between the moment at which a user action is initiated and the time such an action is taken into account by the stage performer or content creation engine. In the context of gaming, this is the time between the moment the user interacts with the game and the moment at which the game engine processes the player's response.
• End-to-End Latency (EEL): The latency for an action that is originally presented in the scene or captured by camera until its visibility on the remote display.
• Round-trip Interaction Delay (RID): The time of an action by the user until it sees the action reflected on its screen. This delay is the sum of the user interaction delay and End-to-End Latency.

The maximum RID depends on the type of scene. A typical example is the Stadia cloud gaming platform and an excellent introduction is provided here [52]. Some extracted high-level requirements on user experience for RID are provided between time 700 to 800 ms [52]. Similar data is collected in clause 4.5 of TR 26.928. Typically, systems have maximum delay requirements between 60ms and 500ms. In terms of formats and bitrates, similar considerations as for clause 6.2.6 apply. However, note that in many cases a pre-rendering is applied in the network, such that data rates and formats are more similar to the split-rendering considerations. Similar considerations as for clause 6.2.6 apply on raster-based split rendering. For EDGAR-based devices, raster-based split rendering based on clause 6.2.5 of TR 26.928 applies. Similar considerations as for clause 6.2.6 apply. The uplink is predominantly the pose information and user interactions. Data rates are several 100 kbit/s and the latency need to be small in order to not add to the overall target latency.

The list of potential standardization area that has been collected is provided in the following:

• Streaming of immersive scenes with 2D and 3D media formats and objects to STAR-based devices including
  • Low-latency streaming protocols to support latencies in the range between 50 to 500ms, typically using RTP-based real-time streaming based on cloud rendering
  • Scene description format, functionality, and profile as an entry point of immersive media
  • Simplified 3D media formats and 2D media formats with integration for STAR-based devices
  • Relevant subset of media codecs for different media types and formats
  • RTP encapsulation of media formats
  • 5G System and 5G Media Streaming support
• Split rendering delivery of immersive scenes to EDGAR-based devices
  • Simple 2D media formats that match AR glass display capabilities
  • Media payload format to be mapped into RTP streams
  • Capability exchange mechanism and relevant signalling
  • Protocol stack and content delivery protocol
  • Cross-layer design, radio and 5G system optimizations for QoS support
  • Uplink streaming of predicted pose information
• Required QoE metrics