AI:

Artificial Intelligence. In the network domain, AI refers to machine-learning-based technologies for automated network operation and other tasks.

AM:

Alternate Marking. A flow performance measurement method, as specified in [RFC 8321].

BMP:

BGP Monitoring Protocol. Specified in [RFC 7854].

DPI:

Deep Packet Inspection. Refers to the techniques that examine packets beyond packet L3/L4 headers.

gNMI:

gRPC Network Management Interface. A network management protocol from the OpenConfig Operator Working Group, mainly contributed by Google. See [gnmi] for details.

GPB:

Google Protocol Buffer. An extensible mechanism for serializing structured data. See [gpb] for details.

gRPC:

gRPC Remote Procedure Call. An open-source high-performance RPC framework that gNMI is based on. See [grpc] for details.

IPFIX:

IP Flow Information Export Protocol. Specified in [RFC 7011].

IOAM:

[RFC 9197]. A data plane on-path telemetry technique.

JSON:

JavaScript Object Notation. An open standard file format and data interchange format that uses human-readable text to store and transmit data objects, as specified in [RFC 8259].

MIB:

Management Information Base. A database used for managing the entities in a network.

NETCONF:

Network Configuration Protocol. Specified in [RFC 6241].

NetFlow:

A Cisco protocol used for flow record collecting, as described in [RFC 3954].

Network Telemetry:

The process and instrumentation for acquiring and utilizing network data remotely for network monitoring and operation. A general term for a large set of network visibility techniques and protocols, concerning aspects like data generation, collection, correlation, and consumption. Network telemetry addresses current network operation issues and enables smooth evolution toward future intent-driven autonomous networks.

NMS:

Network Management System. Refers to applications that allow network administrators to manage a network.

OAM:

Operations, Administration, and Maintenance. A group of network management functions that provide network fault indication, fault localization, performance information, and data and diagnosis functions. Most conventional network monitoring techniques and protocols belong to network OAM.

PBT:

Postcard-Based Telemetry. A data plane on-path telemetry technique. A representative technique is described in [IPPM-IOAM-DIRECT-EXPORT].

RESTCONF:

An HTTP-based protocol that provides a programmatic interface for accessing data defined in YANG, using the datastore concepts defined in NETCONF, as specified in [RFC 8040].

SMIv2:

Structure of Management Information Version 2. Defines MIB objects, as specified in [RFC 2578].

SNMP:

Simple Network Management Protocol. Versions 1, 2, and 3 are specified in [RFC 1157], [RFC 3416], and [RFC 3411], respectively.

XML:

Extensible Markup Language. A markup language for data encoding that is both human readable and machine readable, as specified by W3C [W3C.REC-xml-20081126].

YANG:

YANG is a data modeling language for the definition of data sent over network management protocols such as NETCONF and RESTCONF. YANG is defined in [RFC 6020] and [RFC 7950].

YANG ECA:

A YANG model for Event-Condition-Action policies, as defined in [NETMOD-ECA-POLICY].

YANG-Push:

A mechanism that allows subscriber applications to request a stream of updates from a YANG datastore on a network device. Details are specified in [RFC 8639] and [RFC 8641].

• Security: Network intrusion detection and prevention systems need to monitor network traffic and activities and act upon anomalies. Given increasingly sophisticated attack vectors coupled with increasingly severe consequences of security breaches, new tools and techniques need to be developed, relying on wider and deeper visibility into networks. The ultimate goal is to achieve security with no, or only minimal, human intervention and without disrupting legitimate traffic flows.
• Policy and Intent Compliance: Network policies are the rules that constrain the services for network access, provide service differentiation, or enforce specific treatment on the traffic. For example, a service function chain is a policy that requires the selected flows to pass through a set of ordered network functions. Intent, as defined in [NMRG-IBN-CONCEPTS-DEFINITIONS], is a set of operational goals that a network should meet and outcomes that a network is supposed to deliver, defined in a declarative manner without specifying how to achieve or implement them. An intent requires a complex translation and mapping process before being applied on networks. While a policy or intent is enforced, the compliance needs to be verified and monitored continuously by relying on visibility that is provided through network telemetry data. Any violation must be reported immediately - this will alert the network administrator to the policy or intent violation and will potentially result in updates to how the policy or intent is applied in the network to ensure that it remains in force.
• SLA Compliance: A Service Level Agreement (SLA) is a service contract between a service provider and a client, which includes the metrics for the service measurement and remedy/penalty procedures when the service level misses the agreement. Users need to check if they get the service as promised, and network operators need to evaluate how they can deliver services that meet the SLA based on real-time network telemetry data, including data from network measurements.
• Root Cause Analysis: Many network failures can be the effect of a sequence of chained events. Troubleshooting and recovery require quick identification of the root cause of any observable issues. However, the root cause is not always straightforward to identify, especially when the failure is sporadic and the number of event messages, both related and unrelated to the same cause, is overwhelming. While technologies such as machine learning can be used for root cause analysis, it is up to the network to sense and provide the relevant diagnostic data that are either actively fed into or passively retrieved by the root cause analysis applications.
• Network Optimization: This covers all short-term and long-term network optimization techniques, including load balancing, Traffic Engineering (TE), and network planning. Network operators are motivated to optimize their network utilization and differentiate services for better Return on Investment (ROI) or lower Capital Expenditure (CAPEX). The first step is to know the real-time network conditions before applying policies for traffic manipulation. In some cases, microbursts need to be detected in a very short time frame so that fine-grained traffic control can be applied to avoid network congestion. Long-term planning of network capacity and topology requires analysis of real-world network telemetry data that is obtained over long periods of time.
• Event Tracking and Prediction: The visibility into traffic path and performance is critical for services and applications that rely on healthy network operation. Numerous related network events are of interest to network operators. For example, network operators want to learn where and why packets are dropped for an application flow. They also want to be warned of issues in advance, so proactive actions can be taken to avoid catastrophic consequences.

• Most use cases need to continuously monitor the network and dynamically refine the data collection in real time. Poll-based low-frequency data collection is ill-suited for these applications. Subscription-based streaming data directly pushed from the data source (e.g., the forwarding chip) is preferred to provide sufficient data quantity and precision at scale.
• Comprehensive data is needed, ranging from packet processing engines to traffic managers, line cards to main control boards, user flows to control protocol packets, device configurations to operations, and physical layers to application layers. Conventional OAM only covers a narrow range of data (e.g., SNMP only handles data from the Management Information Base (MIB)). Classical network devices cannot provide all the necessary probes. More open and programmable network devices are therefore needed.
• Many application scenarios need to correlate network-wide data from multiple sources (i.e., from distributed network devices, different components of a network device, or different network planes). A piecemeal solution is often lacking the capability to consolidate the data from multiple sources. The composition of a complete solution, as partly proposed by [NMRG-ANTICIPATED-ADAPTATION], will be empowered and guided by a comprehensive framework.
• Some conventional OAM techniques (e.g., CLI and Syslog) lack a formal data model. The unstructured data hinder the tool automation and application extensibility. Standardized data models are essential to support the programmable networks.
• Although some conventional OAM techniques support data push (e.g., [RFC 2981][RFC 3877], Syslog, and [RFC 3176]), the pushed data are limited to only predefined management plane warnings (e.g., SNMP Trap) or sampled user packets (e.g., sFlow). Network operators require the data with arbitrary source, granularity, and precision, which is beyond the capability of the existing techniques.
• Conventional passive measurement techniques can either consume excessive network resources and produce excessive redundant data or lead to inaccurate results; on the other hand, conventional active measurement techniques can interfere with the user traffic, and their results are indirect. Techniques that can collect direct and on-demand data from user traffic are more favorable.

• Push and Streaming: Instead of polling data from network devices, telemetry collectors subscribe to streaming data pushed from data sources in network devices.
• Volume and Velocity: Telemetry data is intended to be consumed by machines rather than by human beings. Therefore, the data volume can be huge, and the processing is optimized for the needs of automation in real time.
• Normalization and Unification: Telemetry aims to address the overall network automation needs. Efforts are made to normalize the data representation and unify the protocols, so as to simplify data analysis and provide integrated analysis across heterogeneous devices and data sources across a network.
• Model-Based: Telemetry data is modeled in advance, which allows applications to configure and consume data with ease.
• Data Fusion: The data for a single application can come from multiple data sources (e.g., cross-domain, cross-device, and cross-layer) that are based on a common name/ID and need to be correlated to take effect.
• Dynamic and Interactive: Since the network telemetry means to be used in a closed control loop for network automation, it needs to run continuously and adapt to the dynamic and interactive queries from the network operation controller.

• In-Network Customization: The data that is generated can be customized in network at runtime to cater to the specific need of applications. This needs the support of a programmable data plane, which allows probes with custom functions to be deployed at flexible locations.
• In-Network Data Aggregation and Correlation: Network devices and aggregation points can work out which events and what data needs to be stored, reported, or discarded, thus reducing the load on the central collection and processing points while still ensuring that the right information is ready to be processed in a timely way.
• In-Network Processing: Sometimes it is not necessary or feasible to gather all information to a central point to be processed and acted upon. It is possible for the data processing to be done in network, allowing reactive actions to be taken locally.
• Direct Data Plane Export: The data originated from data plane forwarding chips can be directly exported to the data consumer for efficiency, especially when the data bandwidth is large and real-time processing is required.
• In-Band Data Collection: In addition to the passive and active data collection approaches, the new hybrid approach allows to directly collect data for any target flow on its entire forwarding path [OPSAWG-IFIT-FRAMEWORK].

• Future networks, autonomous or otherwise, depend on holistic and comprehensive network visibility. Use cases and applications are better when supported uniformly and coherently using an integrated, converged mechanism and common telemetry data representations wherever feasible. Therefore, the protocols and mechanisms should be consolidated into a minimum yet comprehensive set. A telemetry framework can help to normalize the technique developments.
• Network visibility presents multiple viewpoints. For example, the device viewpoint takes the network infrastructure as the monitoring object from which the network topology and device status can be acquired, and the traffic viewpoint takes the flows or packets as the monitoring object from which the traffic quality and path can be acquired. An application may need to switch its viewpoint during operation. It may also need to correlate a service and its impact on user experience (UE) to acquire the comprehensive information.
• Applications require network telemetry to be elastic in order to make efficient use of network resources and reduce the impact of processing related to network telemetry on network performance. For example, routine network monitoring should cover the entire network with a low data sampling rate. Only when issues arise or critical trends emerge should telemetry data sources be modified and telemetry data rates be boosted as needed.
• Efficient data aggregation is critical for applications to reduce the overall quantity of data and improve the accuracy of analysis.

                +------------------------------+
                |                              |
                |       Network Operation      |<-------+
                |          Applications        |        |
                |                              |        |
                +------------------------------+        |
                        ^          ^       ^            |
                        |          |       |            |
                        V          V       |            V
                +--------------+-----------|---+  +-----------+
                |              | Control   |   |  |           |
                |              | Plane     |   |  | External  |
                |            <--->         |   |  | Data and  | 
                |              | Telemetry |   |  | Event     |
                |  Management  |       ^   V   |  | Telemetry |
                |  Plane       +-------|-------+  |           |
                |  Telemetry   |       V       |  +-----------+ 
                |              | Forwarding    |  
                |              | Plane         |
                |            <--->             |
                |              | Telemetry     |
                |              |               |
                +--------------+---------------+

• Data Object
• Data Export Location
• Data Model
• Data Encoding
• Telemetry Application Protocol
• Data Transport Method

• Convenient Data Subscription: An application should have the freedom to choose which data is exported (see Section 3.3) and the means and frequency of how that data is exported (e.g., on-change or periodic subscription).
• Structured Data: For automatic network operation, machines will replace humans for network data comprehension. Data modeling languages, such as YANG, can efficiently describe structured data and normalize data encoding and transformation.
• High-Speed Data Transport: In order to keep up with the velocity of information, a data source needs to be able to send large amounts of data at high frequency. Compact encoding formats or data compression schemes are needed to reduce the quantity of data and improve the data transport efficiency. The subscription mode, by replacing the query mode, reduces the interactions between clients and servers and helps to improve the data source's efficiency.
• Network Congestion Avoidance: The application must protect the network from congestion with congestion control mechanisms or, at minimum, with circuit breakers. [RFC 8084] and [RFC 8085] provide some solutions in this space.

• How to correlate the End-to-End (E2E) Key Performance Indicators (KPIs) to a specific layer's KPIs. For example, IPTV users may describe their UE by the video smoothness and definition. Then in case of an unusually poor UE KPI or a service disconnection, it is non-trivial to delimit and pinpoint the issue in the responsible protocol layer (e.g., the transport layer or the network layer), the responsible protocol (e.g., IS-IS or BGP at the network layer), and finally the responsible device(s) with specific reasons.
• Conventional OAM-based approaches for control plane KPI measurement, which include Ping (L3), Traceroute (L3), [y1731] (L2), and so on. One common issue behind these methods is that they only measure the KPIs instead of reflecting the actual running status of these protocols, making them less effective or efficient for control plane troubleshooting and network optimization.
• How more research is needed for the BGP monitoring protocol (BMP). BMP is an example of the control plane telemetry; it is currently used for monitoring BGP routes and enables rich applications, such as BGP peer analysis, Autonomous System (AS) analysis, prefix analysis, and security analysis. However, the monitoring of other layers, protocols, and the cross-layer, cross-protocol KPI correlations are still in their infancy (e.g., IGP monitoring is not as extensive as BMP), which requires further research.

• A data plane device's main function is user traffic processing and forwarding. While supporting network visibility is important, the telemetry is just an auxiliary function, and it should strive to not impede normal traffic processing and forwarding (i.e., the forwarding behavior should not be altered, and the trade-off between forwarding performance and telemetry should be well-balanced).
• Network operation applications require end-to-end visibility across various sources, which can result in a huge volume of data. However, the sheer quantity of data must not exhaust the network bandwidth, regardless of the data delivery approach (i.e., whether through in-band or out-of-band channels).
• The data plane devices must provide timely data with the minimum possible delay. Long processing, transport, storage, and analysis delay can impact the effectiveness of the control loop and even render the data useless.
• The data should be structured, labeled, and easy for applications to parse and consume. At the same time, the data types needed by applications can vary significantly. The data plane devices need to provide enough flexibility and programmability to support the precise data provision for applications.
• The data plane telemetry should support incremental deployment and work even though some devices are unaware of the system.
• The requirement and solutions for network congestion avoidance are also applicable to the forwarding plane telemetry.

• Active, Passive, and Hybrid: This dimension pertains to the end-to-end measurement. Active and passive methods (as well as the hybrid types) are well documented in [RFC 7799]. Passive methods include TCPDUMP, [RFC 7011], sFlow, and traffic mirroring. These methods usually have low data coverage. The bandwidth cost is very high in order to improve the data coverage. On the other hand, active methods include Ping, the [RFC 4656], the [RFC 5357], the [RFC 8762], and [RFC 6812]. These methods are intrusive and only provide indirect network measurements. Hybrid methods, including [RFC 9197], [RFC 8321], and [RFC 8889], provide a well-balanced and more flexible approach. However, these methods are also more complex to implement.
• In-Band and Out-of-Band: Telemetry data carried in user packets before being exported to a data collector is considered in-band (e.g., [RFC 9197]). Telemetry data that is directly exported to a data collector without modifying user packets is considered out-of-band (e.g., the postcard-based approach described in Appendix A.3.5). It is also possible to have hybrid methods, where only the telemetry instruction or partial data is carried by user packets (e.g., [RFC 8321]).
• End-to-End and In-Network: End-to-end methods start from, and end at, the network end hosts (e.g., Ping). In-network methods work in networks and are transparent to end hosts. However, if needed, in-network methods can be easily extended into end hosts.
• Data Subject: Depending on the telemetry objective, the methods can be flow based (e.g., [RFC 9197]), path based (e.g., Traceroute), and node based (e.g., [RFC 7011]). The various data objects can be packet, flow record, measurement, states, and signal.

• The role of the external event detector can be played by multiple elements, including hardware (e.g., physical sensors, such as seismometers) and software (e.g., big data sources that can analyze streams of information, such as Twitter messages). Thus, the transmitted data must support different shapes but, at the same time, follow a common but extensible schema.
• Since the main function of the external event detectors is to perform the notifications, their timeliness is assumed. However, once messages have been dispatched, they must be quickly collected and inserted into the control plane with variable priority, which is higher for important sources and events and lower for secondary ones.
• The schema used by external detectors must be easily adopted by current and future devices and applications. Therefore, it must be easily mapped to current data models, such as in terms of YANG.
• As the communication with external entities outside the boundary of a provider network may be realized over the Internet, the risk of congestion is even more relevant in this context and proper countermeasures must be taken. Solutions such as network transport circuit breakers are needed as well.

• Data Query, Analysis, and Storage: This component works at the network operation application block in Figure 1. It is normally a part of the network management system at the receiver side. On one hand, it is responsible for issuing data requirements. The data of interest can be modeled data through configuration or custom data through programming. The data requirements can be queries for one-shot data or subscriptions for events or streaming data. On the other hand, it receives, stores, and processes the returned data from network devices. Data analysis can be interactive to initiate further data queries. This component can reside in either network devices or remote controllers. It can be centralized and distributed and involve one or more instances.
• Data Configuration and Subscription: This component manages data queries on devices. It determines the protocol and channel for applications to acquire desired data. This component is also responsible for configuring the desired data that might not be directly available from data sources. The subscription data can be described by models, templates, or programs.
• Data Encoding and Export: This component determines how telemetry data is delivered to the data analysis and storage component with access control. The data encoding and the transport protocol may vary due to the data export location.
• Data Generation and Processing: The requested data needs to be captured, filtered, processed, and formatted in network devices from raw data sources. This may involve in-network computing and processing on either the fast path or the slow path in network devices.
• Data Object and Source: This component determines the monitoring objects and original data sources provisioned in the device. A data source usually just provides raw data that needs further processing. Each data source can be considered a probe. Some data sources can be dynamically installed, while others will be more static.

                  +----------------------------------------+
                +----------------------------------------+ |
                |                                        | |
                |    Data Query, Analysis, & Storage     | |
                |                                        | +    
                +-------+++ -----------------------------+    
                        |||                   ^^^               
                        |||                   |||               
                        ||V                   |||                
                     +--+V--------------------+++------------+
                  +-----V---------------------+------------+ |
                +---------------------+-------+----------+ | |
                | Data Configuration  |                  | | |
                | & Subscription      | Data Encoding    | | |  
                | (model, template,   | & Export         | | | 
                |  & program)         |                  | | |
                +---------------------+------------------| | | 
                |                                        | | |
                |           Data Generation              | | |
                |           & Processing                 | | |
                |                                        | | |
                +----------------------------------------| | |
                |                                        | | |
                |       Data Object and Source           | |-+
                |                                        |-+
                +----------------------------------------+

• Simple Data: Data that are steadily available from some datastore or static probes in network devices.
• Derived Data: Data that need to be synthesized or processed in the network from raw data from one or more network devices. The data processing function can be statically or dynamically loaded into network devices.
• Event-triggered Data: Data that are conditionally acquired based on the occurrence of some events. An example of event-triggered data could be an interface changing operational state between up and down. Such data can be actively pushed through subscription or passively polled through query. There are many ways to model events, including using Finite State Machine (FSM) or [NETMOD-ECA-POLICY].
• Streaming Data: Data that are continuously generated. It can be a time series or the dump of databases. For example, an interface packet counter is exported every second. The streaming data reflect real-time network states and metrics and require large bandwidth and processing power. The streaming data are always actively pushed to the subscribers.

   +----------------------+     +-----------------+
   | Event-Triggered Data |<----+ Streaming Data  |
   +-------+---+----------+     +-----+---+-------+
           |   |                      |   |
           |   |                      |   |
           |   |   +--------------+   |   |
           |   +-->| Derived Data |<--+   |
           |       +------+------ +       |
           |              |               |
           |              V               |
           |       +--------------+       |
           +------>| Simple Data  |<------+
                   +--------------+

Stage 0 - Static Telemetry:

The telemetry data source and type are determined at design time. The network operator can only configure how to use it with limited flexibility.

Stage 1 - Dynamic Telemetry:

The custom telemetry data can be dynamically programmed or configured at runtime without interrupting the network operation, allowing a trade-off among resource, performance, flexibility, and coverage.

Stage 2 - Interactive Telemetry:

The network operator can continuously customize and fine tune the telemetry data in real time to reflect the network operation's visibility requirements. Compared with Stage 1, the changes are frequent based on the real-time feedback. At this stage, some tasks can be automated, but human operators still need to sit in the middle to make decisions.

Stage 3 - Closed-Loop Telemetry:

The telemetry is free from the interference of human operators, except for generating the reports. The intelligent network operation engine automatically issues the telemetry data requests, analyzes the data, and updates the network operations in closed control loops.

• Telemetry framework trust and policy models;
• Role management and access control for enabling and disabling telemetry capabilities;
• Protocol transport used for telemetry data and its inherent security capabilities;
• Telemetry data stores, storage encryption, methods of access, and retention practices;
• Tracking telemetry events and any abnormalities that might identify malicious attacks using telemetry interfaces.
• Authentication and integrity protection of telemetry data to make data more trustworthy; and
• Segregating the telemetry data traffic from the data traffic carried over the network (e.g., historically management access and management data may be carried via an independent management network).

[gnmi]

R. Shakir, A. Shaikh, P. Borman, M. Hines, C. Lebsack, and C. Marrow, "gRPC Network Management Interface", March 2017,
<https://datatracker.ietf.org/meeting/98/materials/slides-98-rtgwg-gnmi-intro-draft-openconfig-rtgwg-gnmi-spec-00>.

[gpb]

Google Developers, "Protocol Buffers",
<https://developers.google.com/protocol-buffers>.

[grpc]

gRPC, "gPPC: A high performance, open source universal RPC framework",
<https://grpc.io>.

[IPPM-IOAM-DIRECT-EXPORT]

Huawei, , , , , , , , and , "In-situ OAM Direct Exporting", Internet-Draft draft-ietf-ippm-ioam-direct-export-07, October 2021,
<https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-direct-export-07>.

[IPPM-POSTCARD-BASED-TELEMETRY]

LG U+, , , , , , , , , and , "In-Situ OAM Marking-based Direct Export", Internet-Draft draft-song-ippm-postcard-based-telemetry-12, May 2022,
<https://datatracker.ietf.org/doc/html/draft-song-ippm-postcard-based-telemetry-12>.

[NETCONF-DISTRIB-NOTIF]

INSA-Lyon, , , , , and , "Subscription to Distributed Notifications", Internet-Draft draft-ietf-netconf-distributed-notif-03, January 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-netconf-distributed-notif-03>.

[NETCONF-UDP-NOTIF]

NTT, , , , , , and , "UDP-based Transport for Configured Subscriptions", Internet-Draft draft-ietf-netconf-udp-notif-05, March 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-netconf-udp-notif-05>.

[NETMOD-ECA-POLICY]

Cisco, , , , , and , "A YANG Data model for ECA Policy Management", Internet-Draft draft-ietf-netmod-eca-policy-01, February 2021,
<https://datatracker.ietf.org/doc/html/draft-ietf-netmod-eca-policy-01>.

[NMRG-ANTICIPATED-ADAPTATION]

NICT, , "Exploiting External Event Detectors to Anticipate Resource Requirements for the Elastic Adaptation of SDN/NFV Systems", Internet-Draft draft-pedro-nmrg-anticipated-adaptation-02, June 2018,
<https://datatracker.ietf.org/doc/html/draft-pedro-nmrg-anticipated-adaptation-02>.

[NMRG-IBN-CONCEPTS-DEFINITIONS]

Microsoft, , , , and , "Intent-Based Networking - Concepts and Definitions", Internet-Draft draft-irtf-nmrg-ibn-concepts-definitions-09, March 2022,
<https://datatracker.ietf.org/doc/html/draft-irtf-nmrg-ibn-concepts-definitions-09>.

[OPSAWG-DNP4IQ]

Huawei Technologies Co., Ltd, , and , "Requirements for Interactive Query with Dynamic Network Probes", Internet-Draft draft-song-opsawg-dnp4iq-01, June 2017,
<https://datatracker.ietf.org/doc/html/draft-song-opsawg-dnp4iq-01>.

[OPSAWG-IFIT-FRAMEWORK]

SK Telecom, , , , , and , "A Framework for In-situ Flow Information Telemetry", Internet-Draft draft-song-opsawg-ifit-framework-17, February 2022,
<https://datatracker.ietf.org/doc/html/draft-song-opsawg-ifit-framework-17>.

[RFC1157]

J.D. Case, M. Fedor, M.L. Schoffstall, and J. Davin, "Simple Network Management Protocol (SNMP)", RFC 1157, DOI 10.17487/RFC1157, May 1990,
<https://www.rfc-editor.org/info/rfc1157>.

[RFC2578]

K. McCloghrie, D. Perkins, and J. Schoenwaelder, "Structure of Management Information Version 2 (SMIv2)", STD 58, RFC 2578, DOI 10.17487/RFC2578, April 1999,
<https://www.rfc-editor.org/info/rfc2578>.

[RFC2981]

R. Kavasseri, "Event MIB", RFC 2981, DOI 10.17487/RFC2981, October 2000,
<https://www.rfc-editor.org/info/rfc2981>.

[RFC3176]

P. Phaal, S. Panchen, and N. McKee, "InMon Corporation's sFlow: A Method for Monitoring Traffic in Switched and Routed Networks", RFC 3176, DOI 10.17487/RFC3176, September 2001,
<https://www.rfc-editor.org/info/rfc3176>.

[RFC3411]

D. Harrington, R. Presuhn, and B. Wijnen, "An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks", STD 62, RFC 3411, DOI 10.17487/RFC3411, December 2002,
<https://www.rfc-editor.org/info/rfc3411>.

[RFC3416]

R. Presuhn, "Version 2 of the Protocol Operations for the Simple Network Management Protocol (SNMP)", STD 62, RFC 3416, DOI 10.17487/RFC3416, December 2002,
<https://www.rfc-editor.org/info/rfc3416>.

[RFC3877]

S. Chisholm, and D. Romascanu, "Alarm Management Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877, September 2004,
<https://www.rfc-editor.org/info/rfc3877>.

[RFC3954]

B. Claise, "Cisco Systems NetFlow Services Export Version 9", RFC 3954, DOI 10.17487/RFC3954, October 2004,
<https://www.rfc-editor.org/info/rfc3954>.

[RFC4656]

S. Shalunov, B. Teitelbaum, A. Karp, J. Boote, and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006,
<https://www.rfc-editor.org/info/rfc4656>.

[RFC5085]

T. Nadeau, and C. Pignataro, "Pseudowire Virtual Circuit Connectivity Verification (VCCV): A Control Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085, December 2007,
<https://www.rfc-editor.org/info/rfc5085>.

[RFC5357]

K. Hedayat, R. Krzanowski, A. Morton, K. Yum, and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, DOI 10.17487/RFC5357, October 2008,
<https://www.rfc-editor.org/info/rfc5357>.

[RFC5424]

R. Gerhards, "The Syslog Protocol", RFC 5424, DOI 10.17487/RFC5424, March 2009,
<https://www.rfc-editor.org/info/rfc5424>.

[RFC6020]

M. Bjorklund, "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, DOI 10.17487/RFC6020, October 2010,
<https://www.rfc-editor.org/info/rfc6020>.

[RFC6241]

R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman, "Network Configuration Protocol (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
<https://www.rfc-editor.org/info/rfc6241>.

[RFC6812]

M. Chiba, A. Clemm, S. Medley, J. Salowey, S. Thombare, and E. Yedavalli, "Cisco Service-Level Assurance Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013,
<https://www.rfc-editor.org/info/rfc6812>.

[RFC7011]

B. Claise, B. Trammell, and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", STD 77, RFC 7011, DOI 10.17487/RFC7011, September 2013,
<https://www.rfc-editor.org/info/rfc7011>.

[RFC7258]

S. Farrell, and H. Tschofenig, "Pervasive Monitoring Is an Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May 2014,
<https://www.rfc-editor.org/info/rfc7258>.

[RFC7276]

T. Mizrahi, N. Sprecher, E. Bellagamba, and Y. Weingarten, "An Overview of Operations, Administration, and Maintenance (OAM) Tools", RFC 7276, DOI 10.17487/RFC7276, June 2014,
<https://www.rfc-editor.org/info/rfc7276>.

[RFC7540]

M. Belshe, R. Peon, and M. Thomson, "Hypertext Transfer Protocol Version 2 (HTTP/2)", RFC 7540, DOI 10.17487/RFC7540, May 2015,
<https://www.rfc-editor.org/info/rfc7540>.

[RFC7575]

M. Behringer, M. Pritikin, S. Bjarnason, A. Clemm, B. Carpenter, S. Jiang, and L. Ciavaglia, "Autonomic Networking: Definitions and Design Goals", RFC 7575, DOI 10.17487/RFC7575, June 2015,
<https://www.rfc-editor.org/info/rfc7575>.

[RFC7799]

A. Morton, "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016,
<https://www.rfc-editor.org/info/rfc7799>.

[RFC7854]

J. Scudder, R. Fernando, and S. Stuart, "BGP Monitoring Protocol (BMP)", RFC 7854, DOI 10.17487/RFC7854, June 2016,
<https://www.rfc-editor.org/info/rfc7854>.

[RFC7950]

M. Bjorklund, "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016,
<https://www.rfc-editor.org/info/rfc7950>.

[RFC8040]

A. Bierman, M. Bjorklund, and K. Watsen, "RESTCONF Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017,
<https://www.rfc-editor.org/info/rfc8040>.

[RFC8084]

G. Fairhurst, "Network Transport Circuit Breakers", BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017,
<https://www.rfc-editor.org/info/rfc8084>.

[RFC8085]

L. Eggert, G. Fairhurst, and G. Shepherd, "UDP Usage Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, March 2017,
<https://www.rfc-editor.org/info/rfc8085>.

[RFC8259]

T. Bray, "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.

[RFC8321]

G. Fioccola, A. Capello, M. Cociglio, L. Castaldelli, M. Chen, L. Zheng, G. Mirsky, and T. Mizrahi, "Alternate-Marking Method for Passive and Hybrid Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321, January 2018,
<https://www.rfc-editor.org/info/rfc8321>.

[RFC8639]

E. Voit, A. Clemm, A. Gonzalez Prieto, E. Nilsen-Nygaard, and A. Tripathy, "Subscription to YANG Notifications", RFC 8639, DOI 10.17487/RFC8639, September 2019,
<https://www.rfc-editor.org/info/rfc8639>.

[RFC8641]

A. Clemm, and E. Voit, "Subscription to YANG Notifications for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641, September 2019,
<https://www.rfc-editor.org/info/rfc8641>.

[RFC8671]

T. Evens, S. Bayraktar, P. Lucente, P. Mi, and S. Zhuang, "Support for Adj-RIB-Out in the BGP Monitoring Protocol (BMP)", RFC 8671, DOI 10.17487/RFC8671, November 2019,
<https://www.rfc-editor.org/info/rfc8671>.

[RFC8762]

G. Mirsky, G. Jun, H. Nydell, and R. Foote, "Simple Two-Way Active Measurement Protocol", RFC 8762, DOI 10.17487/RFC8762, March 2020,
<https://www.rfc-editor.org/info/rfc8762>.

[RFC8889]

G. Fioccola, M. Cociglio, A. Sapio, and R. Sisto, "Multipoint Alternate-Marking Method for Passive and Hybrid Performance Monitoring", RFC 8889, DOI 10.17487/RFC8889, August 2020,
<https://www.rfc-editor.org/info/rfc8889>.

[RFC8924]

S. Aldrin, C. Pignataro, N. Kumar, R. Krishnan, and A. Ghanwani, "Service Function Chaining (SFC) Operations, Administration, and Maintenance (OAM) Framework", RFC 8924, DOI 10.17487/RFC8924, October 2020,
<https://www.rfc-editor.org/info/rfc8924>.

[RFC9069]

T. Evens, S. Bayraktar, M. Bhardwaj, and P. Lucente, "Support for Local RIB in the BGP Monitoring Protocol (BMP)", RFC 9069, DOI 10.17487/RFC9069, February 2022,
<https://www.rfc-editor.org/info/rfc9069>.

[RFC9197]

F. Brockners, S. Bhandari, and T. Mizrahi, "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022,
<https://www.rfc-editor.org/info/rfc9197>.

[W3C.REC-xml-20081126]

T. Bray, J. Paoli, M. Sperberg-McQueen, E. Maler, and F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth Edition)", November 2008,
<https://www.w3.org/TR/2008/REC-xml-20081126>.

[y1731]

ITU-T, "Operations, administration and maintenance (OAM) functions and mechanisms for Ethernet-based networks", ITU-T Recommendation G.8013/Y.1731, August 2015,
<https://www.itu.int/rec/T-REC-Y.1731/en>.

• A full-duplex streaming transport model; when combined with a binary encoding mechanism, it provides good telemetry efficiency.
• A higher-level feature consistency across platforms that common HTTP/2 libraries typically do not provide. This characteristic is especially valuable for the fact that telemetry data collectors normally reside on a large variety of platforms.
• A built-in load-balancing and failover mechanism.

• Smart objects and sensors. With the consolidation of the Internet of Things (IoT), any network system will have many smart objects attached to its physical surroundings and logical operation environments. Most of these objects will be essentially based on sensors of many kinds (e.g., temperature, humidity, and presence), and the information they provide can be very useful for the management of the network, even when they are not specifically deployed for such purpose. Elements of this source type will usually provide a specific protocol for interaction, especially one of the protocols related to IoT, such as the Constrained Application Protocol (CoAP).
• Online news reporters. Several online news services have the ability to provide an enormous quantity of information about different events occurring in the world. Some of those events can have an impact on the network system managed by a specific framework; therefore, such information may be of interest to the management solution. For instance, diverse security reports, such as Common Vulnerabilities and Exposures (CVEs), can be issued by the corresponding authority and used by the management solution to update the managed system, if needed. Instead of a specific protocol and data format, the sources of this kind of information usually follow a relaxed but structured format. This format will be part of both the ontology and information model of the telemetry framework.
• Global event analyzers. The advance of big data analyzers provides a huge amount of information and, more interestingly, the identification of events detected by analyzing many data streams from different origins. In contrast with the other types of sources, which are focused on specific events, the detectors of this source type will detect generic events. For example, during a sports event, some unexpected movement makes it fascinating, and many people connect to sites that are reporting on the event. The underlying networks supporting the services that cover the event can be affected by such situation, so their management solutions should be aware of it. In contrast with the other source types, a new information model, format, and reporting protocol is required to integrate the detectors of this type with the management solution.

Haoyu Song

Fengwei Qin

Pedro Martinez-Julia

Laurent Ciavaglia

Aijun Wang