Tech-invite3GPPspaceIETFspace
96959493929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 6184

RTP Payload Format for H.264 Video

Pages: 101
Proposed Standard
Errata
Obsoletes:  3984
Part 1 of 4 – Pages 1 to 10
None   None   Next

Top   ToC   RFC6184 - Page 1
Internet Engineering Task Force (IETF)                        Y.-K. Wang
Request for Comments: 6184                                       R. Even
Obsoletes: 3984                                      Huawei Technologies
Category: Standards Track                                  T. Kristensen
ISSN: 2070-1721                                                 Tandberg
                                                                R. Jesup
                                                WorldGate Communications
                                                                May 2011


                   RTP Payload Format for H.264 Video

Abstract

This memo describes an RTP Payload format for the ITU-T Recommendation H.264 video codec and the technically identical ISO/IEC International Standard 14496-10 video codec, excluding the Scalable Video Coding (SVC) extension and the Multiview Video Coding extension, for which the RTP payload formats are defined elsewhere. The RTP payload format allows for packetization of one or more Network Abstraction Layer Units (NALUs), produced by an H.264 video encoder, in each RTP payload. The payload format has wide applicability, as it supports applications from simple low bitrate conversational usage, to Internet video streaming with interleaved transmission, to high bitrate video-on-demand. This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized in Section 14. Issues on backward compatibility to RFC 3984 are discussed in Section 15. Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6184.
Top   ToC   RFC6184 - Page 2
Copyright Notice

   Copyright (c) 2011 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

1. Introduction ....................................................4 1.1. The H.264 Codec ............................................4 1.2. Parameter Set Concept ......................................5 1.3. Network Abstraction Layer Unit Types .......................6 2. Conventions .....................................................7 3. Scope ...........................................................7 4. Definitions and Abbreviations ...................................7 4.1. Definitions ................................................7 4.2. Abbreviations ..............................................9 5. RTP Payload Format .............................................10 5.1. RTP Header Usage ..........................................10 5.2. Payload Structures ........................................12 5.3. NAL Unit Header Usage .....................................13 5.4. Packetization Modes .......................................16 5.5. Decoding Order Number (DON) ...............................17 5.6. Single NAL Unit Packet ....................................19 5.7. Aggregation Packets .......................................20 5.7.1. Single-Time Aggregation Packet (STAP) ..............22 5.7.2. Multi-Time Aggregation Packets (MTAPs) .............25 5.8. Fragmentation Units (FUs) .................................29 6. Packetization Rules ............................................33 6.1. Common Packetization Rules ................................33 6.2. Single NAL Unit Mode ......................................34 6.3. Non-Interleaved Mode ......................................34 6.4. Interleaved Mode ..........................................34 7. De-Packetization Process .......................................35 7.1. Single NAL Unit and Non-Interleaved Mode ..................35 7.2. Interleaved Mode ..........................................35 7.2.1. Size of the De-Interleaving Buffer .................36 7.2.2. De-Interleaving Process ............................36 7.3. Additional De-Packetization Guidelines ....................38
Top   ToC   RFC6184 - Page 3
   8. Payload Format Parameters ......................................39
      8.1. Media Type Registration ...................................39
      8.2. SDP Parameters ............................................57
           8.2.1. Mapping of Payload Type Parameters to SDP ..........57
           8.2.2. Usage with the SDP Offer/Answer Model ..............58
           8.2.3. Usage in Declarative Session Descriptions ..........66
      8.3. Examples ..................................................68
      8.4. Parameter Set Considerations ..............................75
      8.5. Decoder Refresh Point Procedure Using In-Band
           Transport of Parameter Sets (Informative)..................78
           8.5.1. IDR Procedure to Respond to a Request for
                  a Decoder Refresh Point ............................78
           8.5.2. Gradual Recovery Procedure to Respond to
                  a Request for a Decoder Refresh Point ..............79
   9. Security Considerations ........................................79
   10. Congestion Control ............................................80
   11. IANA Considerations ...........................................81
   12. Informative Appendix: Application Examples ....................81
      12.1. Video Telephony According to Annex A of ITU-T
            Recommendation H.241 .....................................81
      12.2. Video Telephony, No Slice Data Partitioning, No
            NAL Unit Aggregation .....................................82
      12.3. Video Telephony, Interleaved Packetization Using
            NAL Unit Aggregation .....................................82
      12.4. Video Telephony with Data Partitioning ...................83
      12.5. Video Telephony or Streaming with FUs and Forward
            Error Correction .........................................83
      12.6. Low Bitrate Streaming ....................................86
      12.7. Robust Packet Scheduling in Video Streaming ..............86
   13. Informative Appendix: Rationale for Decoding Order Number .....87
      13.1. Introduction .............................................87
      13.2. Example of Multi-Picture Slice Interleaving ..............88
      13.3. Example of Robust Packet Scheduling ......................89
      13.4. Robust Transmission Scheduling of Redundant Coded
            Slices ...................................................93
      13.5. Remarks on Other Design Possibilities ....................94
   14. Changes from RFC 3984 .........................................94
   15. Backward Compatibility to RFC 3984 ............................96
   16. Acknowledgements ..............................................98
   17. References ....................................................98
      17.1. Normative References .....................................98
      17.2. Informative References ...................................99
Top   ToC   RFC6184 - Page 4

1. Introduction

This memo specifies an RTP payload specification for the video coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC International Standard 14496-10 [2] (both also known as Advanced Video Coding (AVC)). In this memo, the name H.264 is used for the codec and the standard, but this memo is equally applicable to the ISO/IEC counterpart of the coding standard. This memo obsoletes RFC 3984. Changes from RFC 3984 are summarized in Section 14. Issues on backward compatibility to RFC 3984 are discussed in Section 15.

1.1. The H.264 Codec

The H.264 video codec has a very broad application range that covers all forms of digital compressed video, from low bitrate Internet streaming applications to HDTV broadcast and Digital Cinema applications with nearly lossless coding. Compared to the current state of technology, the overall performance of H.264 is such that bitrate savings of 50% or more are reported. Digital Satellite TV quality, for example, was reported to be achievable at 1.5 Mbit/s, compared to the current operation point of MPEG 2 video at around 3.5 Mbit/s [10]. The codec specification [1] itself conceptually distinguishes between a Video Coding Layer (VCL) and a Network Abstraction Layer (NAL). The VCL contains the signal processing functionality of the codec; mechanisms such as transform, quantization, and motion-compensated prediction; and a loop filter. It follows the general concept of most of today's video codecs, a macroblock-based coder that uses inter picture prediction with motion compensation and transform coding of the residual signal. The VCL encoder outputs slices: a bit string that contains the macroblock data of an integer number of macroblocks and the information of the slice header (containing the spatial address of the first macroblock in the slice, the initial quantization parameter, and similar information). Macroblocks in slices are arranged in scan order unless a different macroblock allocation is specified using the syntax of slice groups. In-picture prediction is used only within a slice. More information is provided in [10]. The NAL encoder encapsulates the slice output of the VCL encoder into Network Abstraction Layer Units (NALUs), which are suitable for transmission over packet networks or for use in packet-oriented
Top   ToC   RFC6184 - Page 5
   multiplex environments.  Annex B of H.264 defines an encapsulation
   process to transmit such NALUs over bytestream-oriented networks.  In
   the scope of this memo, Annex B is not relevant.

   Internally, the NAL uses NAL units.  A NAL unit consists of a one-
   byte header and the payload byte string.  The header indicates the
   type of the NAL unit, the (potential) presence of bit errors or
   syntax violations in the NAL unit payload, and information regarding
   the relative importance of the NAL unit for the decoding process.
   This RTP payload specification is designed to be unaware of the bit
   string in the NAL unit payload.

   One of the main properties of H.264 is the complete decoupling of the
   transmission time, the decoding time, and the sampling or
   presentation time of slices and pictures.  The decoding process
   specified in H.264 is unaware of time, and the H.264 syntax does not
   carry information such as the number of skipped frames (as is common
   in the form of the Temporal Reference in earlier video compression
   standards).  Also, there are NAL units that affect many pictures and
   that are, therefore, inherently timeless.  For this reason, the
   handling of the RTP timestamp requires some special considerations
   for NAL units for which the sampling or presentation time is not
   defined or, at transmission time, is unknown.

1.2. Parameter Set Concept

One very fundamental design concept of H.264 is to generate self- contained packets, to make mechanisms such as the header duplication of RFC 4629 [11] or MPEG-4 Visual's Header Extension Code (HEC) [12] unnecessary. This was achieved by decoupling information relevant to more than one slice from the media stream. This higher-layer meta information should be sent reliably, asynchronously, and in advance from the RTP packet stream that contains the slice packets. (Provisions for sending this information in-band are also available for applications that do not have an out-of-band transport channel appropriate for the purpose). The combination of the higher-level parameters is called a parameter set. The H.264 specification includes two types of parameter sets: sequence parameter sets and picture parameter sets. An active sequence parameter set remains unchanged throughout a coded video sequence, and an active picture parameter set remains unchanged within a coded picture. The sequence and picture parameter set structures contain information such as picture size, optional coding modes employed, and macroblock to slice group map.
Top   ToC   RFC6184 - Page 6
   To be able to change picture parameters (such as the picture size)
   without having to transmit parameter set updates synchronously to the
   slice packet stream, the encoder and decoder can maintain a list of
   more than one sequence and picture parameter set.  Each slice header
   contains a codeword that indicates the sequence and picture parameter
   set to be used.

   This mechanism allows the decoupling of the transmission of parameter
   sets from the packet stream and the transmission of them by external
   means (e.g., as a side effect of the capability exchange) or through
   a (reliable or unreliable) control protocol.  It may even be possible
   that they are never transmitted but are fixed by an application
   design specification.

1.3. Network Abstraction Layer Unit Types

Tutorial information on the NAL design can be found in [13], [14], and [15]. All NAL units consist of a single NAL unit type octet, which also co-serves as the payload header of this RTP payload format. A description of the payload of a NAL unit follows. The syntax and semantics of the NAL unit type octet are specified in [1], but the essential properties of the NAL unit type octet are summarized below. The NAL unit type octet has the following format: +---------------+ |0|1|2|3|4|5|6|7| +-+-+-+-+-+-+-+-+ |F|NRI| Type | +---------------+ The semantics of the components of the NAL unit type octet, as specified in the H.264 specification, are described briefly below. F: 1 bit forbidden_zero_bit. The H.264 specification declares a value of 1 as a syntax violation. NRI: 2 bits nal_ref_idc. A value of 00 indicates that the content of the NAL unit is not used to reconstruct reference pictures for inter picture prediction. Such NAL units can be discarded without risking the integrity of the reference pictures. Values greater than 00 indicate that the decoding of the NAL unit is required to maintain the integrity of the reference pictures.
Top   ToC   RFC6184 - Page 7
   Type:    5 bits
            nal_unit_type.  This component specifies the NAL unit
            payload type as defined in Table 7-1 of [1] and later within
            this memo.  For a reference of all currently defined NAL
            unit types and their semantics, please refer to Section
            7.4.1 in [1].

   This memo introduces new NAL unit types, which are presented in
   Section 5.2.  The NAL unit types defined in this memo are marked as
   unspecified in [1].  Moreover, this specification extends the
   semantics of F and NRI as described in Section 5.3.

2. Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [4]. This specification uses the notion of setting and clearing a bit when bit fields are handled. Setting a bit is the same as assigning that bit the value of 1 (On). Clearing a bit is the same as assigning that bit the value of 0 (Off).

3. Scope

This payload specification can only be used to carry the "naked" H.264 NAL unit stream over RTP and not the bitstream format discussed in Annex B of H.264. Likely, the first applications of this specification will be in the conversational multimedia field, video telephony or video conferencing, but the payload format also covers other applications, such as Internet streaming and TV over IP.

4. Definitions and Abbreviations

4.1. Definitions

This document uses the definitions of [1]. The following terms, defined in [1], are summed up for convenience: access unit: A set of NAL units always containing a primary coded picture. In addition to the primary coded picture, an access unit may also contain one or more redundant coded pictures or other NAL units not containing slices or slice data partitions of a coded picture. The decoding of an access unit always results in a decoded picture.
Top   ToC   RFC6184 - Page 8
      coded video sequence: A sequence of access units that consists, in
      decoding order, of an instantaneous decoding refresh (IDR) access
      unit followed by zero or more non-IDR access units including all
      subsequent access units up to but not including any subsequent IDR
      access unit.

      IDR access unit: An access unit in which the primary coded picture
      is an IDR picture.

      IDR picture: A coded picture containing only slices with I or SI
      slice types that causes a "reset" in the decoding process.  After
      the decoding of an IDR picture, all following coded pictures in
      decoding order can be decoded without inter prediction from any
      picture decoded prior to the IDR picture.

      primary coded picture: The coded representation of a picture to be
      used by the decoding process for a bitstream conforming to H.264.
      The primary coded picture contains all macroblocks of the picture.

      redundant coded picture: A coded representation of a picture or a
      part of a picture.  The content of a redundant coded picture shall
      not be used by the decoding process for a bitstream conforming to
      H.264.  The content of a redundant coded picture may be used by
      the decoding process for a bitstream that contains errors or
      losses.

      VCL NAL unit: A collective term used to refer to coded slice and
      coded data partition NAL units.

   In addition, the following definitions apply:

      decoding order number (DON): A field in the payload structure or a
      derived variable indicating NAL unit decoding order.  Values of
      DON are in the range of 0 to 65535, inclusive.  After reaching the
      maximum value, the value of DON wraps around to 0.

      NAL unit decoding order: A NAL unit order that conforms to the
      constraints on NAL unit order given in Section 7.4.1.2 in [1].

      NALU-time: The value that the RTP timestamp would have if the NAL
      unit would be transported in its own RTP packet.

      transmission order: The order of packets in ascending RTP sequence
      number order (in modulo arithmetic).  Within an aggregation
      packet, the NAL unit transmission order is the same as the order
      of appearance of NAL units in the packet.
Top   ToC   RFC6184 - Page 9
      media-aware network element (MANE): A network element, such as a
      middlebox or application layer gateway that is capable of parsing
      certain aspects of the RTP payload headers or the RTP payload and
      reacting to the contents.

         Informative note: The concept of a MANE goes beyond normal
         routers or gateways in that a MANE has to be aware of the
         signaling (e.g., to learn about the payload type mappings of
         the media streams) and that it has to be trusted when working
         with Secure Real-time Transport Protocol (SRTP).  The advantage
         of using MANEs is that they allow packets to be dropped
         according to the needs of the media coding.  For example, if a
         MANE has to drop packets due to congestion on a certain link,
         it can identify and remove those packets whose elimination
         produces the least adverse effect on the user experience.

      static macroblock: A certain amount of macroblocks in the video
      stream can be defined as static, as defined in Section 8.3.2.8 in
      [3].  Static macroblocks free up additional processing cycles for
      the handling of non-static macroblocks.  Based on a given amount
      of video processing resources and a given resolution, a higher
      number of static macroblocks enables a correspondingly higher
      frame rate.

      default sub-profile: The subset of coding tools, which may be all
      coding tools of one profile or the common subset of coding tools
      of more than one profile, indicated by the profile-level-id
      parameter.

      default level: The level indicated by the profile-level-id
      parameter, which consists of three octets, profile_idc, profile-
      iop, and level_idc.  The default level is indicated by level_idc
      in most cases, and, in some cases, additionally by profile-iop.

4.2. Abbreviations

DON: Decoding Order Number DONB: Decoding Order Number Base DOND: Decoding Order Number Difference FEC: Forward Error Correction FU: Fragmentation Unit IDR: Instantaneous Decoding Refresh IEC: International Electrotechnical Commission ISO: International Organization for Standardization ITU-T: International Telecommunication Union, Telecommunication Standardization Sector MANE: Media-Aware Network Element MTAP: Multi-Time Aggregation Packet
Top   ToC   RFC6184 - Page 10
      MTAP16:     MTAP with 16-bit timestamp offset
      MTAP24:     MTAP with 24-bit timestamp offset
      NAL:        Network Abstraction Layer
      NALU:       NAL Unit
      SAR:        Sample Aspect Ratio
      SEI:        Supplemental Enhancement Information
      STAP:       Single-Time Aggregation Packet
      STAP-A:     STAP type A
      STAP-B:     STAP type B
      TS:         Timestamp
      VCL:        Video Coding Layer
      VUI:        Video Usability Information



(page 10 continued on part 2)

Next Section