RFC 7011
Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information
3. IPFIX Message Format
An IPFIX Message consists of a Message Header, followed by zero or more Sets. The Sets can be any of these three possible types: Data Set, Template Set, or Options Template Set. The format of the IPFIX Message is shown in Figure B. +----------------------------------------------------+ | Message Header | +----------------------------------------------------+ | Set | +----------------------------------------------------+ | Set | +----------------------------------------------------+ ... +----------------------------------------------------+ | Set | +----------------------------------------------------+ Figure B: IPFIX Message Format
Following are some examples of IPFIX Messages:
1. An IPFIX Message consisting of interleaved Template, Data, and
Options Template Sets, as shown in Figure C. Here, Template and
Options Template Sets are transmitted "on demand", before the
first Data Set whose structure they define.
+--------+--------------------------------------------------------+
| | +----------+ +---------+ +-----------+ +---------+ |
|Message | | Template | | Data | | Options | | Data | |
| Header | | Set | | Set | ... | Template | | Set | |
| | | | | | | Set | | | |
| | +----------+ +---------+ +-----------+ +---------+ |
+--------+--------------------------------------------------------+
Figure C: IPFIX Message: Example 1
2. An IPFIX Message consisting entirely of Data Sets, sent after the
appropriate Template Records have been defined and transmitted to
the Collecting Process, as shown in Figure D.
+--------+----------------------------------------------+
| | +---------+ +---------+ +---------+ |
|Message | | Data | | Data | | Data | |
| Header | | Set | ... | Set | ... | Set | |
| | +---------+ +---------+ +---------+ |
+--------+----------------------------------------------+
Figure D: IPFIX Message: Example 2
3. An IPFIX Message consisting entirely of Template and Options
Template Sets, as shown in Figure E. Such a message can be used
to define or redefine Templates and Options Templates in bulk.
+--------+-------------------------------------------------+
| | +----------+ +----------+ +----------+ |
|Message | | Template | | Template | | Options | |
| Header | | Set | ... | Set | ... | Template | |
| | | | | | | Set | |
| | +----------+ +----------+ +----------+ |
+--------+-------------------------------------------------+
Figure E: IPFIX Message: Example 3
3.1. Message Header Format
The format of the IPFIX Message Header is shown in Figure F. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version Number | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Export Time | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Observation Domain ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure F: IPFIX Message Header Format Each Message Header field is exported in network byte order. The fields are defined as follows: Version Version of IPFIX to which this Message conforms. The value of this field is 0x000a for the current version, incrementing by one the version used in the NetFlow services export version 9 [RFC3954]. Length Total length of the IPFIX Message, measured in octets, including Message Header and Set(s). Export Time Time at which the IPFIX Message Header leaves the Exporter, expressed in seconds since the UNIX epoch of 1 January 1970 at 00:00 UTC, encoded as an unsigned 32-bit integer.
Sequence Number
Incremental sequence counter modulo 2^32 of all IPFIX Data Records
sent in the current stream from the current Observation Domain by
the Exporting Process. Each SCTP Stream counts sequence numbers
separately, while all messages in a TCP connection or UDP session
are considered to be part of the same stream. This value can be
used by the Collecting Process to identify whether any IPFIX Data
Records have been missed. Template and Options Template Records
do not increase the Sequence Number.
Observation Domain ID
A 32-bit identifier of the Observation Domain that is locally
unique to the Exporting Process. The Exporting Process uses the
Observation Domain ID to uniquely identify to the Collecting
Process the Observation Domain that metered the Flows. It is
RECOMMENDED that this identifier also be unique per IPFIX Device.
Collecting Processes SHOULD use the Transport Session and the
Observation Domain ID field to separate different export streams
originating from the same Exporter. The Observation Domain ID
SHOULD be 0 when no specific Observation Domain ID is relevant for
the entire IPFIX Message, for example, when exporting the
Exporting Process Statistics, or in the case of a hierarchy of
Collectors when aggregated Data Records are exported.
3.2. Field Specifier Format
Vendors need the ability to define proprietary Information Elements,
because, for example, they are delivering a pre-standards product, or
the Information Element is in some way commercially sensitive. This
section describes the Field Specifier format for both IANA-registered
Information Elements [IANA-IPFIX] and enterprise-specific Information
Elements.
The Information Elements are identified by the Information Element
identifier. When the Enterprise bit is set to 0, the corresponding
Information Element appears in [IANA-IPFIX], and the Enterprise
Number MUST NOT be present. When the Enterprise bit is set to 1, the
corresponding Information Element identifier identified an
enterprise-specific Information Element; the Enterprise Number MUST
be present. An example of this is shown in Appendix A.2.2.
The Field Specifier format is shown in Figure G. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |E| Information Element ident. | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure G: Field Specifier Format Where: E Enterprise bit. This is the first bit of the Field Specifier. If this bit is zero, the Information Element identifier identifies an Information Element in [IANA-IPFIX], and the four-octet Enterprise Number field MUST NOT be present. If this bit is one, the Information Element identifier identifies an enterprise-specific Information Element, and the Enterprise Number field MUST be present. Information Element identifier A numeric value that represents the Information Element. Refer to [IANA-IPFIX]. Field Length The length of the corresponding encoded Information Element, in octets. Refer to [IANA-IPFIX]. The Field Length may be smaller than that listed in [IANA-IPFIX] if the reduced-size encoding is used (see Section 6.2). The value 65535 is reserved for variable- length Information Elements (see Section 7). Enterprise Number IANA enterprise number [IANA-PEN] of the authority defining the Information Element identifier in this Template Record.
3.3. Set and Set Header Format
A Set is a generic term for a collection of records that have a similar structure. There are three different types of Sets: Template Sets, Options Template Sets, and Data Sets. Each of these Sets consists of a Set Header and one or more records. The Set Format and the Set Header Format are defined in the following sections.3.3.1. Set Format
A Set has the format shown in Figure H. The record types can be either Template Records, Options Template Records, or Data Records. The record types MUST NOT be mixed within a Set. +--------------------------------------------------+ | Set Header | +--------------------------------------------------+ | record | +--------------------------------------------------+ | record | +--------------------------------------------------+ ... +--------------------------------------------------+ | record | +--------------------------------------------------+ | Padding (opt.) | +--------------------------------------------------+ Figure H: Set Format Set Header The Set Header Format is defined in Section 3.3.2. Record One of the record formats: Template Record, Options Template Record, or Data Record format. Padding The Exporting Process MAY insert some padding octets, so that the subsequent Set starts at an aligned boundary. For security reasons, the padding octet(s) MUST be composed of octets with value zero (0). The padding length MUST be shorter than any allowable record in this Set. If padding of the IPFIX Message is desired in combination with very short records, then the padding Information Element 'paddingOctets' can be used for padding
records such that their length is increased to a multiple of 4 or
8 octets. Because Template Sets are always 4-octet aligned by
definition, padding is only needed in the case of other
alignments, e.g., on 8-octet boundaries.
3.3.2. Set Header Format
Every Set contains a common header. This header is defined in
Figure I.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure I: Set Header Format
Each Set Header field is exported in network format. The fields are
defined as follows:
Set ID
Identifies the Set. A value of 2 is reserved for Template Sets.
A value of 3 is reserved for Options Template Sets. Values from 4
to 255 are reserved for future use. Values 256 and above are used
for Data Sets. The Set ID values of 0 and 1 are not used, for
historical reasons [RFC3954].
Length
Total length of the Set, in octets, including the Set Header, all
records, and the optional padding. Because an individual Set MAY
contain multiple records, the Length value MUST be used to
determine the position of the next Set.
3.4. Record Format
IPFIX defines three record formats, as defined in the next sections: the Template Record format, the Options Template Record format, and the Data Record format.3.4.1. Template Record Format
One of the essential elements in the IPFIX record format is the Template Record. Templates greatly enhance the flexibility of the record format because they allow the Collecting Process to process IPFIX Messages without necessarily knowing the interpretation of all Data Records. A Template Record contains any combination of IANA- assigned and/or enterprise-specific Information Element identifiers. The format of the Template Record is shown in Figure J. It consists of a Template Record Header and one or more Field Specifiers. Field Specifiers are defined in Figure G above. +--------------------------------------------------+ | Template Record Header | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ ... +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ Figure J: Template Record Format The format of the Template Record Header is shown in Figure K. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID (> 255) | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure K: Template Record Header Format
The Template Record Header Field definitions are as follows:
Template ID
Each Template Record is given a unique Template ID in the range
256 to 65535. This uniqueness is local to the Transport Session
and Observation Domain that generated the Template ID. Since
Template IDs are used as Set IDs in the Sets they describe (see
Section 3.4.3), values 0-255 are reserved for special Set types
(e.g., Template Sets themselves), and Templates and Options
Templates (see Section 3.4.2) cannot share Template IDs within a
Transport Session and Observation Domain. There are no
constraints regarding the order of the Template ID allocation. As
Exporting Processes are free to allocate Template IDs as they see
fit, Collecting Processes MUST NOT assume incremental Template
IDs, or anything about the contents of a Template based on its
Template ID alone.
Field Count
Number of fields in this Template Record.
The example in Figure L shows a Template Set with mixed IANA-assigned and enterprise-specific Information Elements. It consists of a Set Header, a Template Header, and several Field Specifiers. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| Information Element id. 1.1 | Field Length 1.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number 1.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| Information Element id. 1.2 | Field Length 1.2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| Information Element id. 1.N | Field Length 1.N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number 1.N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| Information Element id. 2.1 | Field Length 2.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| Information Element id. 2.2 | Field Length 2.2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number 2.2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| Information Element id. 2.M | Field Length 2.M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number 2.M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Padding (opt) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure L: Template Set Example Information Element id.s 1.2 and 2.1 appear in [IANA-IPFIX] (Enterprise bit = 0) and therefore do not need an Enterprise Number to identify them.
3.4.2. Options Template Record Format
Thanks to the notion of scope, The Options Template Record gives the Exporter the ability to provide additional information to the Collector that would not be possible with Flow Records alone. See Section 4 for specific Options Templates used for reporting metadata about IPFIX Exporting and Metering Processes.3.4.2.1. Scope
The scope, which is only available in the Options Template Set, gives the context of the reported Information Elements in the Data Records. The scope is one or more Information Elements, specified in the Options Template Record. At a minimum, Collecting Processes SHOULD support as scope the observationDomainId, exportingProcessId, meteringProcessId, templateId, lineCardId, exporterIPv4Address, exporterIPv6Address, and ingressInterface Information Elements. The IPFIX protocol doesn't prevent the use of any Information Elements for scope. However, some Information Element types don't make sense if specified as scope (for example, the counter Information Elements). The IPFIX Message Header already contains the Observation Domain ID. If not zero, this Observation Domain ID can be considered as an implicit scope for the Data Records in the IPFIX Message. Multiple Scope Fields MAY be present in the Options Template Record, in which case the composite scope is the combination of the scopes. For example, if the two scopes are meteringProcessId and templateId, the combined scope is this Template for this Metering Process. If a different order of Scope Fields would result in a Record having a different semantic meaning, then the order of Scope Fields MUST be preserved by the Exporting Process. For example, in the context of PSAMP [RFC5476], if the first scope defines the filtering function, while the second scope defines the sampling function, the order of the scope is important. Applying the sampling function first, followed by the filtering function, would lead to potentially different Data Records than applying the filtering function first, followed by the sampling function.
3.4.2.2. Options Template Record Format
An Options Template Record contains any combination of IANA-assigned and/or enterprise-specific Information Element identifiers. The format of the Options Template Record is shown in Figure M. It consists of an Options Template Record Header and one or more Field Specifiers. Field Specifiers are defined in Figure G above. +--------------------------------------------------+ | Options Template Record Header | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ ... +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ Figure M: Options Template Record Format The format of the Options Template Record Header is shown in Figure N. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID (> 255) | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure N: Options Template Record Header Format
The Options Template Record Header Field definitions are as follows:
Template ID
Each Options Template Record is given a unique Template ID in the
range 256 to 65535. This uniqueness is local to the Transport
Session and Observation Domain that generated the Template ID.
Since Template IDs are used as Set IDs in the sets they describe
(see Section 3.4.3), values 0-255 are reserved for special Set
types (e.g., Template Sets themselves), and Templates and Options
Templates cannot share Template IDs within a Transport Session and
Observation Domain. There are no constraints regarding the order
of the Template ID allocation. As Exporting Processes are free to
allocate Template IDs as they see fit, Collecting Processes MUST
NOT assume incremental Template IDs, or anything about the
contents of an Options Template based on its Template ID alone.
Field Count
Number of all fields in this Options Template Record, including
the Scope Fields.
Scope Field Count
Number of scope fields in this Options Template Record. The Scope
Fields are normal Fields, except that they are interpreted as
scope at the Collector. A scope field count of N specifies that
the first N Field Specifiers in the Template Record are Scope
Fields. The Scope Field Count MUST NOT be zero.
The example in Figure O shows an Options Template Set with mixed IANA-assigned and enterprise-specific Information Elements. It consists of a Set Header, an Options Template Header, and several Field Specifiers. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 258 | Field Count = N + M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field Count = N |0| Scope 1 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length |0| Scope 2 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 2 Field Length | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... |1| Scope N Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope N Field Length | Scope N Enterprise Number ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ... Scope N Enterprise Number |1| Option 1 Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option 1 Field Length | Option 1 Enterprise Number ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ... Option 1 Enterprise Number | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... |0| Option M Infor. Element id. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option M Field Length | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure O: Options Template Set Example
3.4.3. Data Record Format
The Data Records are sent in Data Sets. The format of the Data Record is shown in Figure P. It consists only of one or more Field Values. The Template ID to which the Field Values belong is encoded in the Set Header field "Set ID", i.e., "Set ID" = "Template ID". +--------------------------------------------------+ | Field Value | +--------------------------------------------------+ | Field Value | +--------------------------------------------------+ ... +--------------------------------------------------+ | Field Value | +--------------------------------------------------+ Figure P: Data Record Format Note that Field Values do not necessarily have a length of 16 bits. Field Values are encoded according to their data type as specified in [RFC7012]. Interpretation of the Data Record format can be done only if the Template Record corresponding to the Template ID is available at the Collecting Process.
The example in Figure Q shows a Data Set. It consists of a Set Header and several Field Values. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = Template ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 1 | Record 1 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 1 | Record 2 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Field Value 1 | Record 3 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure Q: Data Set, Containing Data Records4. Specific Reporting Requirements
Some specific Options Templates and Options Template Records are necessary to provide extra information about the Flow Records and about the Metering Process. The Options Template and Options Template Records defined in these subsections, which impose some constraints on the Metering Process and Exporting Process implementations, MAY be implemented. If implemented, the specific Options Templates SHOULD be implemented as specified in these subsections. The minimum set of Information Elements is always specified in these Specific IPFIX Options Templates. Nevertheless, extra Information Elements may be used in these specific Options Templates. The Collecting Process MUST check the possible combinations of Information Elements within the Options Template Records to correctly interpret the following Options Templates.
4.1. The Metering Process Statistics Options Template
The Metering Process Statistics Options Template specifies the structure of a Data Record for reporting Metering Process statistics. It SHOULD contain the following Information Elements, as defined in [IANA-IPFIX]: (scope) observationDomainId This Information Element MUST be defined as a Scope Field and MUST be present, unless the Observation Domain ID of the enclosing Message is non-zero. (scope) meteringProcessId If present, this Information Element MUST be defined as a Scope Field. exportedMessageTotalCount exportedFlowRecordTotalCount exportedOctetTotalCount The Exporting Process SHOULD export the Data Record specified by the Metering Process Statistics Options Template on a regular basis or based on some export policy. This periodicity or export policy SHOULD be configurable. Note that if several Metering Processes are available on the Exporter Observation Domain, the Information Element meteringProcessId MUST be specified as an additional Scope Field.4.2. The Metering Process Reliability Statistics Options Template
The Metering Process Reliability Statistics Options Template specifies the structure of a Data Record for reporting lack of reliability in the Metering Process. It SHOULD contain the following Information Elements, as defined in [IANA-IPFIX]: (scope) observationDomainId This Information Element MUST be defined as a Scope Field and MUST be present, unless the Observation Domain ID of the enclosing Message is non-zero.
(scope) meteringProcessId
If present, this Information Element MUST be defined as a Scope
Field.
ignoredPacketTotalCount
ignoredOctetTotalCount
time first packet ignored
The timestamp of the first packet that was ignored by the
Metering Process. For this timestamp, any of the following
timestamp Information Elements can be used:
observationTimeSeconds,
observationTimeMilliseconds,
observationTimeMicroseconds, or
observationTimeNanoseconds.
time last packet ignored
The timestamp of the last packet that was ignored by the
Metering Process. For this timestamp, any of the following
timestamp Information Elements can be used:
observationTimeSeconds,
observationTimeMilliseconds,
observationTimeMicroseconds, or
observationTimeNanoseconds.
The Exporting Process SHOULD export the Data Record specified by the
Metering Process Reliability Statistics Options Template on a regular
basis or based on some export policy. This periodicity or export
policy SHOULD be configurable.
Note that if several Metering Processes are available on the Exporter
Observation Domain, the Information Element meteringProcessId MUST be
specified as an additional Scope Field.
Since the Metering Process Reliability Statistics Options Template
contains two identical timestamp Information Elements, and since the
order of the Information Elements in the Template Records is not
guaranteed, the Collecting Process interprets the time interval of
ignored packets as the range between the two values; see Section 5.2
for wraparound considerations.
4.3. The Exporting Process Reliability Statistics Options Template
The Exporting Process Reliability Statistics Options Template specifies the structure of a Data Record for reporting lack of reliability in the Exporting Process. It SHOULD contain the following Information Elements, as defined in [IANA-IPFIX]: (scope) Exporting Process Identifier The identifier of the Exporting Process for which reliability is reported. Any of the exporterIPv4Address, exporterIPv6Address, or exportingProcessId Information Elements can be used for this field. This Information Element MUST be defined as a Scope Field. notSentFlowTotalCount notSentPacketTotalCount notSentOctetTotalCount time first flow dropped The time at which the first Flow Record was dropped by the Exporting Process. For this timestamp, any of the following timestamp Information Elements can be used: observationTimeSeconds, observationTimeMilliseconds, observationTimeMicroseconds, or observationTimeNanoseconds. time last flow dropped The time at which the last Flow Record was dropped by the Exporting Process. For this timestamp, any of the following timestamp Information Elements can be used: observationTimeSeconds, observationTimeMilliseconds, observationTimeMicroseconds, or observationTimeNanoseconds. The Exporting Process SHOULD export the Data Record specified by the Exporting Process Reliability Statistics Options Template on a regular basis or based on some export policy. This periodicity or export policy SHOULD be configurable.
Since the Exporting Process Reliability Statistics Options Template contains two identical timestamp Information Elements, and since the order of the Information Elements in the Template Records is not guaranteed, the Collecting Process interprets the time interval of dropped packets as the range between the two values; see Section 5.2 for wraparound considerations.4.4. The Flow Keys Options Template
The Flow Keys Options Template specifies the structure of a Data Record for reporting the Flow Keys of reported Flows. A Flow Keys Data Record extends a particular Template Record that is referenced by its templateId. The Template Record is extended by specifying which of the Information Elements contained in the corresponding Data Records describe Flow properties that serve as Flow Keys of the reported Flow. The Flow Keys Options Template SHOULD contain the following Information Elements, as defined in [IANA-IPFIX]: (scope) templateId This Information Element MUST be defined as a Scope Field. flowKeyIndicator5. Timing Considerations
5.1. IPFIX Message Header Export Time and Flow Record Time
The IPFIX Message Header Export Time field is the time at which the IPFIX Message Header leaves the Exporter, using the same encoding as the dateTimeSeconds abstract data type [RFC7012], i.e., expressed in seconds since the UNIX epoch, 1 January 1970 at 00:00 UTC, encoded as an unsigned 32-bit integer. Certain time-related Information Elements may be expressed as an offset from this Export Time. For example, Data Records requiring a microsecond precision can export the flow start and end times with the flowStartMicroseconds and flowEndMicroseconds Information Elements, which encode the absolute time in microseconds in terms of the NTP epoch, 1 January 1900 at 00:00 UTC, in a 64-bit field. An alternate solution is to export the flowStartDeltaMicroseconds and flowEndDeltaMicroseconds Information Elements in the Data Record, which respectively report the flow start and end time as negative offsets from the Export Time, as an unsigned 32-bit integer. This latter solution lowers the export bandwidth requirement, saving four bytes per timestamp, while increasing the load on the Exporter,
as the Exporting Process must calculate the flowStartDeltaMicroseconds and flowEndDeltaMicroseconds of every single Data Record before exporting the IPFIX Message. It must be noted that timestamps based on the Export Time impose some time constraints on the Data Records contained within the IPFIX Message. In the example of flowStartDeltaMicroseconds and flowEndDeltaMicroseconds Information Elements, the Data Record can only contain records with timestamps within 71 minutes of the Export Time. Otherwise, the 32-bit counter would not be sufficient to contain the flow start time offset.5.2. Supporting Timestamp Wraparound
The dateTimeSeconds abstract data type [RFC7012] and the Export Time Message Header field (Section 3.1) are encoded as 32-bit unsigned integers, expressed as seconds since the UNIX epoch, 1 January 1970 at 00:00 UTC, as defined in [POSIX.1]. These values will wrap around on 7 February 2106 at 06:28:16 UTC. In order to support continued use of the IPFIX protocol beyond this date, Exporting Processes SHOULD export dateTimeSeconds values and the Export Time Message Header field as the number of seconds since the UNIX epoch, 1 January 1970 at 00:00 UTC, modulo 2^32. Collecting Processes SHOULD use the current date, or other contextual information, to properly interpret dateTimeSeconds values and the Export Time Message Header field. There are similar considerations for the NTP-based dateTimeMicroseconds and dateTimeNanoseconds abstract data types [RFC7012]. Exporting Processes SHOULD export dateTimeMicroseconds and dateTimeNanoseconds values as if the NTP era [RFC5905] is implicit; Collecting Processes SHOULD use the current date, or other contextual information, to determine the NTP era in order to properly interpret dateTimeMicroseconds and dateTimeNanoseconds values in received Data Records. The dateTimeMilliseconds abstract data type will wrap around in approximately 500 billion years; the specification of the behavior of this abstract data type after that time is left as a subject of a future revision of this specification. The long-term storage of files [RFC5655] for archival purposes is affected by timestamp wraparound, as the use of the current date to interpret timestamp values in files stored on the order of multiple decades in the past may lead to incorrect values; therefore, it is RECOMMENDED that such files be stored with contextual information to assist in the interpretation of these timestamps.
6. Linkage with the Information Model
As with values in the IPFIX Message Header and Set Header, values of all Information Elements [RFC7012], except for those of the string and octetArray data types, are encoded in canonical format in network byte order (also known as big-endian byte ordering).6.1. Encoding of IPFIX Data Types
The following sections define the encoding of the data types specified in [RFC7012].6.1.1. Integral Data Types
Integral data types -- unsigned8, unsigned16, unsigned32, unsigned64, signed8, signed16, signed32, and signed64 -- MUST be encoded using the default canonical format in network byte order. Signed integral data types are represented in two's complement notation.6.1.2. Address Types
Address types -- macAddress, ipv4Address, and ipv6Address -- MUST be encoded the same way as the integral data types, as six, four, and sixteen octets in network byte order, respectively.6.1.3. float32
The float32 data type MUST be encoded as an IEEE binary32 floating point type as specified in [IEEE.754.2008], in network byte order as specified in Section 3.6 of [RFC1014]. Note that on little-endian machines, byte swapping of the native representation is necessary before export. Note that the method for doing this may be implementation platform dependent.6.1.4. float64
The float64 data type MUST be encoded as an IEEE binary64 floating point type as specified in [IEEE.754.2008], in network byte order as specified in Section 3.7 of [RFC1014]. Note that on little-endian machines, byte swapping of the native representation is necessary before export. Note that the method for doing this may be implementation platform dependent.
6.1.5. boolean
The boolean data type is specified according to the TruthValue in [RFC2579]. It is encoded as a single-octet integer per Section 6.1.1, with the value 1 for true and value 2 for false. Every other value is undefined.6.1.6. string and octetArray
The "string" data type represents a finite-length string of valid characters of the Unicode character encoding set. The string data type MUST be encoded in UTF-8 [RFC3629] format. The string is sent as an array of zero or more octets using an Information Element of fixed or variable length. IPFIX Exporting Processes MUST NOT send IPFIX Messages containing ill-formed UTF-8 string values for Information Elements of the string data type; Collecting Processes SHOULD detect and ignore such values. See [UTF8-EXPLOIT] for background on this issue. The octetArray data type has no encoding rules; it represents a raw array of zero or more octets, with the interpretation of the octets defined in the Information Element definition.6.1.7. dateTimeSeconds
The dateTimeSeconds data type is an unsigned 32-bit integer in network byte order containing the number of seconds since the UNIX epoch, 1 January 1970 at 00:00 UTC, as defined in [POSIX.1]. dateTimeSeconds is encoded identically to the IPFIX Message Header Export Time field. It can represent dates between 1 January 1970 and 7 February 2106 without wraparound; see Section 5.2 for wraparound considerations.6.1.8. dateTimeMilliseconds
The dateTimeMilliseconds data type is an unsigned 64-bit integer in network byte order containing the number of milliseconds since the UNIX epoch, 1 January 1970 at 00:00 UTC, as defined in [POSIX.1]. It can represent dates beginning on 1 January 1970 and for approximately the next 500 billion years without wraparound.6.1.9. dateTimeMicroseconds
The dateTimeMicroseconds data type is a 64-bit field encoded according to the NTP Timestamp format as defined in Section 6 of [RFC5905]. This field is made up of two unsigned 32-bit integers in network byte order: Seconds and Fraction. The Seconds field is the number of seconds since the NTP epoch, 1 January 1900 at 00:00 UTC.
The Fraction field is the fractional number of seconds in units of 1/(2^32) seconds (approximately 233 picoseconds). It can represent dates between 1 January 1900 and 8 February 2036 in the current NTP era; see Section 5.2 for wraparound considerations. Note that dateTimeMicroseconds and dateTimeNanoseconds share an identical encoding. The dateTimeMicroseconds data type is intended only to represent timestamps of microsecond precision. Therefore, the bottom 11 bits of the Fraction field SHOULD be zero and MUST be ignored for all Information Elements of this data type (as 2^11 x 233 picoseconds = .477 microseconds).6.1.10. dateTimeNanoseconds
The dateTimeNanoseconds data type is a 64-bit field encoded according to the NTP Timestamp format as defined in Section 6 of [RFC5905]. This field is made up of two unsigned 32-bit integers in network byte order: Seconds and Fraction. The Seconds field is the number of seconds since the NTP epoch, 1 January 1900 at 00:00 UTC. The Fraction field is the fractional number of seconds in units of 1/(2^32) seconds (approximately 233 picoseconds). It can represent dates between 1 January 1900 and 8 February 2036 in the current NTP era; see Section 5.2 for wraparound considerations. Note that dateTimeMicroseconds and dateTimeNanoseconds share an identical encoding. There is no restriction on the interpretation of the Fraction field for the dateTimeNanoseconds data type.6.2. Reduced-Size Encoding
Information Elements encoded as signed, unsigned, or float data types MAY be encoded using fewer octets than those implied by their type in the information model definition, based on the assumption that the smaller size is sufficient to carry any value the Exporter may need to deliver. This reduces the network bandwidth requirement between the Exporter and the Collector. Note that the Information Element definitions [IANA-IPFIX] always define the maximum encoding size. For instance, the information model defines octetDeltaCount as an unsigned64 type, which would require 64 bits. However, if the Exporter will never locally encounter the need to send a value larger than 4294967295, it may choose to send the value as unsigned32 instead. This behavior is indicated by the Exporter by specifying a size in the Template with a smaller length than that associated with the assigned type of the Information Element. In the example above, the Exporter would place a length of 4 versus 8 in the Template.
Reduced-size encoding MAY be applied to the following integer types: unsigned64, signed64, unsigned32, signed32, unsigned16, and signed16. The signed versus unsigned property of the reported value MUST be preserved. The reduction in size can be to any number of octets smaller than the original type if the data value still fits, i.e., so that only leading zeroes are dropped. For example, an unsigned64 can be reduced in size to 7, 6, 5, 4, 3, 2, or 1 octet(s). Reduced-size encoding MAY be used to reduce float64 to float32. The float32 not only has a reduced number range but, due to the smaller mantissa, is also less precise. In this case, the float64 would be reduced in size to 4 octets. Reduced-size encoding MUST NOT be applied to any other data type defined in [RFC7012] that implies a fixed length, as these types either have internal structure (such as ipv4Address or dateTimeMicroseconds) or restricted ranges that are not suitable for reduced-size encoding (such as dateTimeMilliseconds). Information Elements of type octetArray and string may be exported using any length, subject to restrictions on length specific to each Information Element, as noted in that Information Element's description.7. Variable-Length Information Element
The IPFIX Template mechanism is optimized for fixed-length Information Elements [RFC7012]. Where an Information Element has a variable length, the following mechanism MUST be used to carry the length information for both the IANA-assigned and enterprise-specific Information Elements. In the Template Set, the Information Element Field Length is recorded as 65535. This reserved length value notifies the Collecting Process that the length value of the Information Element will be carried in the Information Element content itself.
In most cases, the length of the Information Element will be less than 255 octets. The following length-encoding mechanism optimizes the overhead of carrying the Information Element length in this more common case. The length is carried in the octet before the Information Element, as shown in Figure R. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Length (< 255)| Information Element | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... continuing as needed | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure R: Variable-Length Information Element (IE) (Length < 255 Octets) The length may also be encoded into 3 octets before the Information Element, allowing the length of the Information Element to be greater than or equal to 255 octets. In this case, the first octet of the Length field MUST be 255, and the length is carried in the second and third octets, as shown in Figure S. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 255 | Length (0 to 65535) | IE | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... continuing as needed | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure S: Variable-Length Information Element (IE) (Length 0 to 65535 Octets) The octets carrying the length (either the first or the first three octets) MUST NOT be included in the length of the Information Element.
(page 38 continued on part 3)
