Content for TR 26.818 Word version: 15.0.0

3.1 Definitions 3.2 Abbreviations 4.1 Test 1 4.2 Test 2 4.3 Test 3
...

0 Introduction p. 7

In the context of the 3GPP activity on the definition of Audio media profiles for Virtual Reality (VR) streaming services, several candidate solutions have been considered. The submission process of Audio media profiles included various tests characterizing the solution performances. The present document presents the detailed test results provided by the solution proponents and the crosscheck labs. The results were not compiled to any merit figures of the candidate solutions hence no such merit figures were used in the selection of the profile. The test results provided in the present document were not generated to be used in direct comparison.

1 Scope p. 8

The present document is a collection of test results on candidate audio media profiles for VR streaming services over 3GPP. A brief description of the 3 tests characterizing the audio media profile performances is presented under clause 4. The following clauses provide the test results from the audio media profile proponent as well as those from the crosscheck labs. The four documented solutions are (in alphabetical order):

DTS-UHD
Metadata Assisted EVS Codec (MAEC)
OMAF 3D Audio Baseline Profile
Spatial AAC extension (spAACe)

2 References p. 8

The following documents contain provisions which, through reference in this text, constitute provisions of the present document.

References are either specific (identified by date of publication, edition number, version number, etc.) or non specific.
For a specific reference, subsequent revisions do not apply.
For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP document (including a GSM document), a non-specific reference implicitly refers to the latest version of that document in the same Release as the present document.

[1]

TR 21.905: "Vocabulary for 3GPP Specifications".

[2]

TS 26.259: "Subjective test methodologies for the evaluation of immersive audio systems".

[3]

ITU-R Recommendation BS.1534-3: "Method for the subjective assessment of intermediate quality level of audio systems".

[4]

ETSI TS 103 491 (V1.1.1): "DTS-UHD Audio Format; Delivery of Channels, Objects and Ambisonic Sound Fields".

[5]

ETSI TS 103 584 (V1.1.1): "DTS-UHD Point Source Renderer".

[6]

ITU-R Recommendation BS.1116-3: "Methods for the subjective assessment of small impairments in audio systems".

[7]

3GPP S4-180835: "Dolby VRStream audio profile candidate - Description of Bitstream, Decoder, and Renderer plus informative Encoder Description". (Document attached to the present document).

[8]

ITU-R Recommendation BS.2051-1: "Advanced sound system for programme production".

[9]

TS 26.118: "3GPP Virtual reality profiles for streaming applications".

[10]

"Google Resonance Monitoring" plugin: (https://github.com/resonance-audio/resonance-audio-draw-tools).

[11]

ISO/IEC 23008-3:2015: "Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 3: 3D audio".

[12]

ISO/IEC 23008-3:2015/Amd 3:2017; "MPEG-H 3D Audio Phase 2".

[13]

SADIE KU100 HRTF database: (https://www.york.ac.uk/sadie-project/database.html).

[14]

3GPP S4-180977: "OMAF 3D Audio Baseline Media Profile for VRStream". (Document attached to the present document).

[15]

TS 26.260: (V0.0.4): "Objective test methodologies for the evaluation of immersive audio systems".

[16]

Pulkki, Ville. "Virtual Sound Source Positioning Using Vector Base Amplitude Panning.": Journal of the Audio Engineering Society 45, no. 6 (June 1, 1997): 456-66.

[17]

Brinkmann, F. and Weinzierl, S. (2017). AKtools - An Open Software Toolbox for Signal Acquisition, Processing, and Inspection in Acoustics: presented at the 142nd AES Convention. Berlin, Germany.

3 Definitions and abbreviations p. 9

3.1 Definitions p. 9

For the purposes of the present document, the terms and definitions given in TR 21.905 and the following apply. A term defined in the present document takes precedence over the definition of the same term, if any, in TR 21.905.

Group of frames:

an interval between two consecutive sync frames

3.2 Abbreviations p. 9

For the purposes of the present document, the abbreviations given in TR 21.905 and the following apply. An abbreviation defined in the present document takes precedence over the definition of the same abbreviation, if any, in TR 21.905.

ART

Artefacts

BAQ

Basic Audio Quality

CCR

Comparison Category Rating

Confidence interval

CIBR

Common Informative Binaural Renderer

Decibel

ESD

Equivalent Spatial Domain

EVS

Enhanced Voice Services

FOA

First Order Ambisonics

GBR

Generic Binaural Renderer

GoF

Group of Frames

HATS

Head and torso simulator

HIQ

High Quality

HOA

Higher Order Ambisonics

Hidden Reference

HRIR

Head-Related Impulse Response

HRTF

Head-Related Transfer Function

ISO BMFF

ISO Based Media File Format

kbps

kilobits per second

LEV

Loudness level

LFE

Low Frequency Effects

Low pass

MUSHRA

MUltiple Stimuli with Hidden Reference and Anchor

OMAF

Omnidirectional Media Application Format

OSC

Open Sound Control

PCM

Pulse Code Modulation

SPA

Spatial quality

SuT

System under Test

TIM

Timbre

VBAP

Vector base amplitude panning

VBR

Variable Bit-Rate

Virtual Reality

VST

Virtual Studio Technology

4 VR streaming tests description p. 10

4.1 Test 1 p. 10

The Test 1 assesses the Codec Quality Characterization for the audio profiles submitted in the context of VR streaming services. The Test 1 Codec Quality Characterization test methodology is defined in clause 5 of TS 26.259. This test assesses the Basic Audio Quality attribute at different bit-rates for a given audio profile.

The audio profile is tested:

according to clause 5 of TS 26.259,
using its own Reference Renderer for Reference and Degraded conditions,
over loudspeakers,
based on ITU-R BS.1534-3 [3] (MUSHRA),
evaluating the Basic Audio Quality.

With 20 test items as input, Test 1 is split into two tests, Test 1a and Test 1b with 10 test items each.

If not included in the original Test 1a and Test 1b, a third test focused on First Order Ambisonics (FOA) called Test 1c was provided. This FOA test only considered the pure HOA source materials that were converted into FOA by truncation to the 1st order.

4.2 Test 2 p. 10

The Test 2 also assesses the Codec Quality Characterization for the audio profiles submitted in the context of VR streaming services but, unlike Test 1, in the binaural environment. The Test 2 Codec Quality Characterization test methodology is defined in TS 26.259, clause 7. This test also assesses the Basic Audio Quality attribute at different bit-rates for a given audio profile.

The audio profile is tested:

according to TS 26.259, clause 7,
using a Common Informative Binaural Renderer (CIBR) for reference and degraded conditions,
over headphones,
based on ITU-R BS.1534-3 [3] (MUSHRA),
evaluating the Basic Audio Quality.

With the same 20 input test items as for Test 1, Test 2 is split into two tests, Test 2a and Test 2b with the same test items as Test 1a and Test 1b respectively.

4.3 Test 3 p. 10

The Test 3 assesses the Reference Binaural Renderer Quality Characterization for the audio profiles submitted in the context of VR streaming services. The Test 3 Reference Binaural Renderer Quality Characterization test methodology is defined in clause 6 of TS 26.259. This test characterizes the audio profile performance with Reference Renderer and, optionally but strongly recommended, the audio profile performance with Common Informative Binaural Renderer.

The audio profile is tested:

according to clause 6 of TS 26.259,
over headphones with head tracking,
evaluating the following attributes:
Spatial Quality
Artefacts
Timbre Quality
Overall Quality

Test 3 contains 12 test items. Test 3 is split into two sessions: One comparing the candidate audio profile with CIBR 1st order, and the other comparing the candidate audio profile with CIBR 3rd order.