Ideally, decoding and rendering of head-trackable immersive audio would be implementable and operational on any UE including XR End Devices like AR glasses or earbuds. However, lightweight End Devices of this class frequently operate under strict constraints especially in terms of computational complexity. Reasons are tight limits in terms of power consumption to reduce battery weight, power dissipation heat, and strict implementation cost constraints. Thus, it may not be possible to always ensure immersive decoding and rendering support consuming the native immersive audio format of a given codec in a lightweight XR End Device. It is thus the objective of Immersive Audio Split Rendering to solve this problem with solutions targeting:

• Complexity of operation in end-rendering lightweight device is reduced substantially compared to the native decoding and head-tracked binaural rendering of the original coded audio format.
• Memory consumption in end-rendering lightweight device is reduced compared to the native decoding and head-tracked binaural rendering of the original coded audio format.
• Minimum impact on QoS/QoE:
  • All required immersive audio formats of the original coding format, coding modes, and operating ranges (bit rates) ought to be supported.
  • Given that Split Rendering relies on pre-rendering on a capable first UE or network node to an intermediate representation, followed by coding and transmitting that representation for decoding and rendering on the lightweight device, it is unavoidably a transcoding approach. The transcoding impact on QoS/QoE, i.e., on quality and latency, thus ought to be as small as possible in comparison to a relevant reference system, which is to operate decoding and head-tracked rendering of the original coding format. This system is referred to as native decoding/rendering reference or for the specific IVAS reference IVAS Local Decoding/Rendering Reference.
  • Head-tracked immersive audio attributes and especially DOF attributes of the audio formats of the original coding format ought to be retained. I.e., if the immersive audio of the original coding format is head-trackable in 3-DOF, Split Rendering ought to retain this property.

The 'golden' reference system for any Split Rendering solution is to operate decoding and head-tracked rendering of the original coding format in the lightweight End Device. This system is referred to as native coding reference or native decoding/rendering reference. This is also the reference considered by the ISAR WID, where the reference to be considered should be the one where audio decoding and rendering happen entirely on the End Device. In clause 4.2.2 the term "Local Audio Rendering" is used for this case. The characteristics of such a solution depend on the specific coding solution, i.e., the "complexity and memory as well as constraints related to relevant interfaces between presentation engine and End Device such as bit rate, latency, down- and upstream traffic characteristics" would be defined by the specific solution. With the IVAS candidate now available, those parameters can also be extracted, using this solution as the reference. This specific reference is referred to as "IVAS Local Decoding/Rendering Reference". A further relevant reference system is a basic transcoding-based system with decoding and head-tracked binaural rendering of the native codec format carried out by the capable UE or network node. The rendered binaural audio signal is subsequently re-encoded. In clause 4.2.4 the term "Remote Audio Rendering" is used for this case. The most viable coding mode of the native codec that can code the binaural audio signal is assumed. Most viable would generally mean a coding mode with least complexity and memory footprint for decoding on the lightweight End Device. This would typically be a stereo coding mode of that codec. Subsequently, the re-encoded binaural audio signal is transmitted to the lightweight End Device where it is decoded and output without final pose adjustment. As the End Device does not carry out any pose corrections matching the actual pose of the End Device, this reference configuration is referred to as 0-DOF native transcoding reference system. This approach is the only reasonable baseline for lightweight End Devices to render binaural audio derived from the native codec format if full native decoding with head-tracked binaural rendering is not possible on that device, implementing the most basic split rendering approach.

Distributed and Remote Rendering are inherently architectures that involve transcoding, i.e. there is an immersive audio representation that is terminated in the Remote MAF and then processed to have an intermediate representation on the link between capable device and lightweight device to achieve a similar QoE as in the case of Local Rendering but allowing implementation on devices with less capabilities. For this, the link characteristics must be considered, such as maximum allowed bit rate on the channel and link latency. Since ISAR solutions need to cope with a range of traffic characteristics (with link-specific devices potentially having individual constraints), it is evident that no single set of requirements would be able to target all scenarios.

Physical attributes are complexity, memory consumption, algorithmic delay, motion-to-sound latency, bit rate, etc. The following generic physical design constraints are provided as an informative guidance when defining the physical design constraints for split rendering solutions applicable to a specific target codec/system:

Functional design constraints are related to the Split Rendering objective that the required immersive audio formats, operation modes and ranges of the original (native) coding format be supported. A further functional attribute associated with immersive audio to be retained is head-trackability. Accordingly, the following generic functional design constraints are provided as an informative guidance when defining the functional design constraints for split rendering solutions applicable to a specific target codec/system:

ISAR supports immersive audio experiences in various use cases. A number of factors can contribute to general QoE of the ISAR solution. These include general audio quality, motion-to-sound latency, end-to-end-latency, spatial image quality, pose accuracy, etc. Relevant reference systems for audio quality requirements are the native coding reference system and the 0-DOF native transcoding reference system. The quality of the native coding reference system is the optimum which cannot be surpassed by a Split Rendering system. The 0-DOF native transcoding reference system on the other hand may suffer from quality degradations due to transcoding and due to potential differences between the assumed End Device pose during binaural rendering and the actual End Device pose. These deviations may be caused by the transmission round trip delay between the End Device and the capable device/network node performing the head-tracked binaural rendering. It is expected that an ISAR solution employing post-rendering with pose correction provides higher quality of experience (QoE) than the 0-DOF native transcoding reference system under a given scenario with significant latency between the End Device and the capable device/network node performing the head-tracked binaural rendering. Given the large number of potential split rendering scenarios, the performance requirements may be defined for one or several relevant scenario(s). In addition, the requirements are expected to be met under the defined physical and functional design constraints. The ISAR solution ought further to provide QoE that is as close as possible to the QoE offered by the native coding reference system under the same scenario.