Content for TR 26.857 Word version: 18.0.0

4.3 External Specifications 4.3.1 General 4.3.2 W3C HTML-5 APIs for Media 4.3.2.1 W3C HTML-5 APIs 4.3.2.2 W3C Media Capabilities framework 4.3.3 Khronos OpenXR 4.3.4 MPEG Network-Based Media Processing (NBMP) media function description
...

4.3 External Specifications p. 16

4.3.1 General p. 16

This clause provides an overview of existing specifications that provide specifications similar to what is envisioned as Media Service Enabler specifications.

4.3.2 W3C HTML-5 APIs for Media p. 16

According to the W3C Process Document [17], W3C Working Groups create specifications and guidelines to complete the scope of work envisioned by a Working Group's charter. These technical reports undergo cycles of revision and review as they advance towards W3C Recommendation status. Once review suggests the Working Group has met their requirements for a new standard, including wide review, a Candidate Recommendation phase allows the Working Group to formally collect implementation experience to demonstrate that the specification works in practice. At the end of the process, the Advisory Committee reviews the mature technical report, and if there is support from its Membership, W3C publishes it as a Recommendation.

In summary, the W3C Recommendation Track consists of:

Publication of the First Public Working Draft.
Publication of zero or more revised Working Drafts.
Publication of one or more Candidate Recommendations.
Publication of a Proposed Recommendation.
Publication as a W3C Recommendation.

A key issue in the process is "adequate implementation experience" that is summarized as follows in [17]. Implementation experience is required to show that a specification is sufficiently clear, complete, and relevant to market needs, to ensure that independent interoperable implementations of each feature of the specification will be realized. While no exhaustive list of requirements is provided here, when assessing that there is adequate implementation experience the following may be considered:

is each feature of the current specification implemented, and how is this demonstrated?
are there independent interoperable implementations of the current specification?
are there implementations created by people other than the authors of the specification?
are implementations publicly deployed?
is there implementation experience at all levels of the specification's ecosystem (authoring, consuming, publishing…)?
are there reports of difficulties or problems with implementation?

Planning and accomplishing a demonstration of (interoperable) implementations can be very time consuming. Groups are often able to work more effectively if they plan how they will demonstrate interoperable implementations early in the development process; for example, developing tests in concert with implementation efforts.

A set of developer tools for W3C specification development is provided here: https://www.w3.org/developers/tools/.

For media streaming media consumption using web-based end points, CTA WAVE has developed several specifications that define and develop test regimes on how to use web-based APIs. For this purpose, requirements and related tests are defined for media playback APIs in particular in CTA WAVE 5000 [18], the WAVE Web Media API Snapshot, and CTA WAVE 5003 [19], the Device Playback Capabilities Specification provide detailed requirements and are supported by test suites for user agent and device tests.

4.3.2.2 W3C Media Capabilities framework p. 17

The W3C Media Capabilities API [10] is designed to query a W3C user agent (such as a web browser) with regard to its media decoding and encoding capabilities. The intent is to provide a replacement for the HTML5 canPlayType() method and the Media Source Extension isTypeSupported() method that more accurately reflects the user agent's media encoding and decoding capabilities.

The API supports decoding from a file, from Media Source Extension (MSE), or from WebRTC, and also encoding to media chunks that can be stored in a file or transmitted as a WebRTC stream.

The video and audio configuration objects define the general characteristics of the video and audio configuration.
The key system configuration object defines the key configuration for the encrypted media (EME).

For a given encoding or decoding configuration, the API returns a binary flag for each of the following values: supported, smooth, or power-efficient. It seems the logic is very similar to canPlayType(), but the configuration can be described in more detail and the result is more accurate.

The specification is characterised as follows:

It limits the expression of capabilities to video and audio encoding/decoding and key configuration for decryption. It doesn't support other media types such as subtitles, images, or graphics.
It doesn't define a general language to define the capabilities of various media functions other than video/audio decoding.
It seems various encoding configurations (e.g. motion estimation, bit rate control modes, number of reference frames, etc) cannot be described or set as the desired configuration.
The API for querying the functionality i still based on a binary pass/fail result (similar to canPlayType()), presumably to reduce the risk of fingerprinting the device, and therefore, it is not possible to get an expressive list of supported features.

4.3.3 Khronos OpenXR p. 17

OpenXR [9] is an API (Application Programming Interface) for XR applications. It sits between an application and an in-process or out-of-process "XR runtime system" (just "runtime" hereafter). The runtime may handle such functionality as frame composition, peripheral management, and raw tracking information.

The OpenXR specification is intended to satisfy the needs of both programmers and runtime implementors:

To the application programmer, OpenXR is a set of functions that interface with a runtime to perform commonly required operations such as accessing controller/peripheral state, getting current and/or predicted tracking positions, and submitting rendered frames.
To the runtime implementor, OpenXR is a set of functions that control the operation of the XR system and establishes the lifecycle of an XR application.

However, the specification does not necessarily provide a model for implementation. A runtime implementation is expected to produce results conforming with those produced by the specified methods, but may carry out particular procedures in ways that are more efficient than the one specified.

A few fundamentals on API definitions:

Version numbers 64 bit major, minor, patch.
Threading, multiprocessing, runtime.
Extensions.
API Layering, which means that a user or application may insert API layers between the application and the runtime implementation.
Return codes.
Handles and Object Handling.
Timing.
Colors.
Coordinate Systems.
Events.

The specification is also supported by API Reference Pages [9] which are generated by automatic extraction from the specification source and document commands, object handles, structures, enumerations, flags, other types and all extensions.

For each command, the following information is provided:

Name.
C function call specification.
Parameter descriptions.
Valid usage.
Return codes.
Reference to OpenXR specification.

OpenXR is an extensible API and the optional functions are call extensions. Since the extensions are options, only a subset of runtimes may implement a particular extension. Therefore, an application may first query which extensions are available from its underlying runtime. Open XR permits extensions to be defined by OpenXR or by external parties. The function xrEnumerateInstanceExtensionProperties [9] provides the list of available extensions in the current runtime. Each extension is identified by a unique name and an extension version, and by the type of the extension.

For the specification itself, the document sources are marked up in Asciidoctor format (https://asciidoctor.org/). Asciidoctor and related toolchain components are used to generate the output documents. A full overview of how the specification is developed is provided here: https://github.com/KhronosGroup/OpenXR-Docs/blob/main/specification/README.md.

Khronos has also released a Conformance Test Suite for OpenXR, published the tests as open source software on GitHub (https://github.com/KhronosGroup/OpenXR-CTS), and launched the OpenXR 1.0 Adopters Program so that implementations can be officially conformant for the first time. Any OpenXR implementer - Khronos member or not - is welcome to become an OpenXR Adopter and submit conformance test results for Working Group review and approval. If their implementation is approved as conformant, they will be able to use the OpenXR trademark on their implementation.

4.3.4 MPEG Network-Based Media Processing (NBMP) media function description p. 18

ISO/IEC 23090-8 [2] is a specification for describing media functions and workflows as microservices running in the cloud. The specification defines the data structures for describing workflows and methods for managing them. The framework enables dynamic creation of media processing pipelines, as well as access to processed media data and metadata in real time or in a deferred way. The media and metadata formats used between the media source, workflow manager, and media processing entities in a media processing pipeline are also specified.

Each function in a media processing pipeline is specified using a JSON object called a function description that describes the functionality, the inputs, outputs, processing characteristics, and configuration parameters, as well as the processing requirements. The function description can also describe events, and even how to set up notifications, reporting, and monitoring for that function if it is deployed as a microservice. More complex functions can be described also as a composite of simpler functions. Finally, the function description JSON object can also be used programmatically to instantiate a task or microservice that performs the function it describes.

The NBMP function description has the advantage of describing being able to describe a function with identifiers as well as a detailed description of a function. The optional parameters of an implementation of a generic function can be described using this approach.