RFC 5810
Forwarding and Control Element Separation (ForCES) Protocol Specification
4. Overview
The reader is referred to the framework document [RFC3746], and in particular, Sections 3 and 4, for an architectural overview and an explanation of how the ForCES protocol fits in. There may be some content overlap between the framework document and this section in order to provide clarity. This document is authoritative on the protocol, whereas [RFC3746] is authoritative on the architecture.
4.1. Protocol Framework
Figure 1 below is reproduced from the framework document for clarity. It shows an NE with two CEs and two FEs. --------------------------------------- | ForCES Network Element | -------------- Fc | -------------- -------------- | | CE Manager |---------+-| CE 1 |------| CE 2 | | -------------- | | | Fr | | | | | -------------- -------------- | | Fl | | | Fp / | | | Fp| |----------| / | | | | |/ | | | | | | | | | Fp /|----| | | | | /--------/ | | -------------- Ff | -------------- -------------- | | FE Manager |---------+-| FE 1 | Fi | FE 2 | | -------------- | | |------| | | | -------------- -------------- | | | | | | | | | | | ----+--+--+--+----------+--+--+--+----- | | | | | | | | | | | | | | | | Fi/f Fi/f Fp: CE-FE interface Fi: FE-FE interface Fr: CE-CE interface Fc: Interface between the CE manager and a CE Ff: Interface between the FE manager and an FE Fl: Interface between the CE manager and the FE manager Fi/f: FE external interface Figure 1: ForCES Architectural Diagram The ForCES protocol domain is found in the Fp reference points. The Protocol Element configuration reference points, Fc and Ff, also play a role in the booting up of the ForCES protocol. The protocol element configuration (indicated by reference points Fc, Ff, and Fl in [RFC3746]) is out of scope of the ForCES protocol but is touched on in this document in discussion of FEM and CEM since it is an integral part of the protocol pre-association phase.
Figure 2 below shows further breakdown of the Fp interfaces by means of the example of an MPLS QoS-enabled Network Element. ------------------------------------------------- | | | | | | | |OSPF |RIP |BGP |RSVP |LDP |. . . | | | | | | | | ------------------------------------------------- CE | ForCES Interface | ------------------------------------------------- ^ ^ | | ForCES | |data control | |packets messages| |(e.g., routing packets) | | v v ------------------------------------------------- | ForCES Interface | ------------------------------------------------- FE | | | | | | | |LPM Fwd|Meter |Shaper |MPLS |Classi-|. . . | | | | | |fier | | ------------------------------------------------- Figure 2: Examples of CE and FE Functions The ForCES interface shown in Figure 2 constitutes two pieces: the PL and the TML.
This is depicted in Figure 3 below. +------------------------------------------------ | CE PL | +------------------------------------------------ | CE TML | +------------------------------------------------ ^ | ForCES | (i.e., ForCES data + control PL | packets ) messages | over | specific | TML | encaps | and | transport | | v +------------------------------------------------ | FE TML | +------------------------------------------------ | FE PL | +------------------------------------------------ Figure 3: ForCES Interface The PL is in fact the ForCES protocol. Its semantics and message layout are defined in this document. The TML layer is necessary to connect two ForCES PLs as shown in Figure 3 above. The TML is out of scope for this document but is within scope of ForCES. This document defines requirements the PL needs the TML to meet. Both the PL and the TML are standardized by the IETF. While only one PL is defined, different TMLs are expected to be standardized. To interoperate, the TML at the CE and FE are expected to conform to the same definition. On transmit, the PL delivers its messages to the TML. The local TML delivers the message to the destination TML. On receive, the TML delivers the message to its destination PL.4.1.1. The PL
The PL is common to all implementations of ForCES and is standardized by the IETF as defined in this document. The PL is responsible for associating an FE or CE to an NE. It is also responsible for tearing
down such associations. An FE uses the PL to transmit various subscribed-to events to the CE PL as well as to respond to various status requests issued from the CE PL. The CE configures both the FE and associated LFBs' operational parameters using the PL. In addition, the CE may send various requests to the FE to activate or deactivate it, reconfigure its HA parameterization, subscribe to specific events, etc. More details can be found in Section 7.4.1.2. The TML
The TML transports the PL messages. The TML is where the issues of how to achieve transport-level reliability, congestion control, multicast, ordering, etc. are handled. It is expected that more than one TML will be standardized. The various possible TMLs could vary their implementations based on the capabilities of underlying media and transport. However, since each TML is standardized, interoperability is guaranteed as long as both endpoints support the same TML. All ForCES protocol layer implementations MUST be portable across all TMLs, because all TMLs MUST have the top-edge semantics defined in this document.4.1.3. The FEM/CEM Interface
The FEM and CEM components, although valuable in the setup and configurations of both the PL and TML, are out of scope of the ForCES protocol. The best way to think of them is as configurations/ parameterizations for the PL and TML before they become active (or even at runtime based on implementation). In the simplest case, the FE or CE reads a static configuration file. RFC 3746 has a more detailed description on how the FEM and CEM could be used. The pre- association phase, where the CEM and FEM can be used, are described briefly in Section 4.2.1. An example of typical things the FEM/CEM could configure would be TML-specific parameterizations such as: a. How the TML connection should happen (for example, what IP addresses to use, transport modes, etc.) b. The ID for the FE (FEID) or CE (CEID) (which would also be issued during the pre-association phase) c. Security parameterization such as keys, etc. d. Connection association parameters
An example of connection association parameters might be:
o simple parameters: send up to 3 association messages every 1
second
o complex parameters: send up to 4 association messages with
increasing exponential timeout
4.2. ForCES Protocol Phases
ForCES, in relation to NEs, involves two phases: the pre-association
phase where configuration/initialization/bootup of the TML and PL
layer happens, and the post-association phase where the ForCES
protocol operates to manipulate the parameters of the FEs.
CE sends Association Setup
+---->--->------------>---->---->---->------->----+
| Y
^ |
| Y
+---+-------+ +-------------+
|FE pre- | | FE post- |
|association| CE sends Association Teardown | association |
|phase |<------- <------<-----<------<-------+ phase |
| | | |
+-----------+ +-------------+
^ Y
| |
+-<---<------<-----<------<----<---------<------+
FE loses association
Figure 4: The FE Protocol Phases
In the mandated case, once associated, the FE may forward packets
depending on the configuration of its specific LFBs. An FE that is
associated to a CE will continue sending packets until it receives an
Association Teardown Message or until it loses association. An
unassociated FE MAY continue sending packets when it has a high
availability capability. The extra details are explained in
Section 8 and not discussed here to allow for a clear explanation of
the basics.
The FE state transitions are controlled by means of the FE Object LFB
FEState component, which is defined in [RFC5812], Section 5.1, and
also explained in Section 7.3.2.
The FE initializes in the FEState OperDisable. When the FE is ready to process packets in the data path, it transitions itself to the OperEnable state. The CE may decide to pause the FE after it already came up as OperEnable. It does this by setting the FEState to AdminDisable. The FE stays in the AdminDisable state until it is explicitly configured by the CE to transition to the OperEnable state. When the FE loses its association with the CE, it may go into the pre-association phase depending on the programmed policy. For the FE to properly complete the transition to the AdminDisable state, it MUST stop packet forwarding and this may impact multiple LFBS. How this is achieved is outside the scope of this specification.4.2.1. Pre-association
The ForCES interface is configured during the pre-association phase. In a simple setup, the configuration is static and is typically read from a saved configuration file. All the parameters for the association phase are well known after the pre-association phase is complete. A protocol such as DHCP may be used to retrieve the configuration parameters instead of reading them from a static configuration file. Note, this will still be considered static pre- association. Dynamic configuration may also happen using the Fc, Ff, and Fl reference points (refer to [RFC3746]). Vendors may use their own proprietary service discovery protocol to pass the parameters. Essentially, only guidelines are provided here and the details are left to the implementation. The following are scenarios reproduced from the framework document to show a pre-association example.
<----Ff ref pt---> <--Fc ref pt-------> FE Manager FE CE Manager CE | | | | | | | | (security exchange) (security exchange) 1|<------------>| authentication 1|<----------->|authentication | | | | (FE ID, components) (CE ID, components) 2|<-------------| request 2|<------------|request | | | | 3|------------->| response 3|------------>|response (corresponding CE ID) (corresponding FE ID) | | | | | | | | Figure 5: Examples of a Message Exchange over the Ff and Fc Reference Points <-----------Fl ref pt--------------> | FE Manager FE CE Manager CE | | | | | | | | (security exchange) | | 1|<------------------------------>| | | | | | (a list of CEs and their components) | 2|<-------------------------------| | | | | | (a list of FEs and their components) | 3|------------------------------->| | | | | | | | | | Figure 6: Example of a Message Exchange over the Fl Reference Point Before the transition to the association phase, the FEM will have established contact with a CEM component. Initialization of the ForCES interface will have completed, and authentication as well as capability discovery may be complete. Both the FE and CE would have the necessary information for connecting to each other for configuration, accounting, identification, and authentication purposes. To summarize, at the completion of this stage both sides have all the necessary protocol parameters such as timers, etc. The Fl reference point may continue to operate during the association phase and may be used to force a disassociation of an FE or CE. The specific interactions of the CEM and the FEM that are part of the
pre-association phase are out of scope; for this reason, these details are not discussed any further in this specification. The reader is referred to the framework document [RFC3746] for a slightly more detailed discussion.4.2.2. Post-association
In this phase, the FE and CE components communicate with each other using the ForCES protocol (PL over TML) as defined in this document. There are three sub-phases: o Association Setup Stage o Established Stage o Association Lost Stage4.2.2.1. Association Setup Stage
The FE attempts to join the NE. The FE may be rejected or accepted. Once granted access into the NE, capabilities exchange happens with the CE querying the FE. Once the CE has the FE capability information, the CE can offer an initial configuration (possibly to restore state) and can query certain components within either an LFB or the FE itself. More details are provided in Section 4.4. On successful completion of this stage, the FE joins the NE and is moved to the Established Stage.4.2.2.2. Established Stage
In this stage, the FE is continuously updated or queried. The FE may also send asynchronous event notifications to the CE or synchronous heartbeat notifications if programmed to do so. This stage continues until a termination occurs, either due to loss of connectivity or due to a termination initiated by either the CE or the FE. Refer to the section on protocol scenarios, Section 4.4, for more details.4.2.2.3. Association Lost Stage
In this stage, both or either the CE or FE declare the other side is no longer associated. The disconnection could be initiated by either party for administrative purposes but may also be driven by operational reasons such as loss of connectivity.
A core LFB known as the FE Protocol Object (FEPO) is defined (refer to Appendix B and Section 7.3.1). FEPO defines various timers that can be used in conjunction with a traffic-sensitive heartbeat mechanism (Section 4.3.3) to detect loss of connectivity. The loss of connectivity between TMLs does not indicate a loss of association between respective PL layers. If the TML cannot repair the transport loss before the programmed FEPO timer thresholds associated with the FE is exceeded, then the association between the respective PL layers will be lost. FEPO defines several policies that can be programmed to define behavior upon a detected loss of association. The FEPO's programmed CE failover policy (refer to Sections 8, 7.3.1, 4.3.3, and B) defines what takes place upon loss of association. For this version of the protocol (as defined in this document), the FE, upon re-association, MUST discard any state it has as invalid and retrieve new state. This approach is motivated by a desire for simplicity (as opposed to efficiency).4.3. Protocol Mechanisms
Various semantics are exposed to the protocol users via the PL header including transaction capabilities, atomicity of transactions, two- phase commits, batching/parallelization, high availability, and failover as well as command pipelines. The EM (Execution Mode) flag, AT (Atomic Transaction) flag, and TP (Transaction Phase) flag as defined in the common header (Section 6.1) are relevant to these mechanisms.4.3.1. Transactions, Atomicity, Execution, and Responses
In the master-slave relationship, the CE instructs one or more FEs on how to execute operations and how to report the results. This section details the different modes of execution that a CE can order the FE(s) to perform, as defined in Section 4.3.1.1. It also describes the different modes a CE can ask the FE(s) to use for formatting the responses after processing the operations as requested. These modes relate to the transactional two-phase commit operations.
4.3.1.1. Execution
There are 3 execution modes that can be requested for a batch of operations spanning one or more LFB selectors (refer to Section 7.1.5) in one protocol message. The EM flag defined in the common header (Section 6.1) selects the execution mode for a protocol message, as below: a. execute-all-or-none b. continue-execute-on-failure c. execute-until-failure4.3.1.1.1. execute-all-or-none
When set to this mode of execution, independent operations in a message MAY be targeted at one or more LFB selectors within an FE. All these operations are executed serially, and the FE MUST have no execution failure for any of the operations. If any operation fails to execute, then all the previous ones that have been executed prior to the failure will need to be undone. That is, there is rollback for this mode of operation. Refer to Section 4.3.1.2.2 for how this mode is used in cases of transactions. In such a case, no operation is executed unless a commit is issued by the CE. Care should be taken on how this mode is used because a mis- configuration could result in traffic losses. To add certainty to the success of an operation, one should use this mode in a transactional operation as described in Section 4.3.1.2.24.3.1.1.2. continue-execute-on-failure
If several independent operations are targeted at one or more LFB selectors, execution continues for all operations at the FE even if one or more operations fail.4.3.1.1.3. execute-until-failure
In this mode, all operations are executed on the FE sequentially until the first failure. The rest of the operations are not executed but operations already completed are not undone. That is, there is no rollback in this mode of operation.
4.3.1.2. Transaction and Atomicity
4.3.1.2.1. Transaction Definition
A transaction is defined as a collection of one or more ForCES operations within one or more PL messages that MUST meet the ACIDity properties [ACID], defined as: Atomicity: In a transaction involving two or more discrete pieces of information, either all of the pieces are committed or none are. Consistency: A transaction either creates a new and valid state of data or, if any failure occurs, returns all data to the state it was in before the transaction was started. Isolation: A transaction in process and not yet committed MUST remain isolated from any other transaction. Durability: Committed data is saved by the system such that, even in the event of a failure and a system restart, the data is available in its correct state. There are cases where the CE knows exact memory and implementation details of the FE such as in the case of an FE-CE pair from the same vendor where the FE-CE pair is tightly coupled. In such a case, the transactional operations may be simplified further by extra computation at the CE. This view is not discussed further other than to mention that it is not disallowed. As defined above, a transaction is always atomic and MAY be a. Within an FE alone Example: updating multiple tables that are dependent on each other. If updating one fails, then any that were already updated MUST be undone. b. Distributed across the NE Example: updating table(s) that are inter-dependent across several FEs (such as L3 forwarding-related tables).4.3.1.2.2. Transaction Protocol
By use of the execution mode, as defined in Section 4.3.1.1, the protocol has provided a mechanism for transactional operations within one stand-alone message. The 'execute-all-or-none' mode can meet the ACID requirements.
For transactional operations of multiple messages within one FE or across FEs, a classical transactional protocol known as two-phase commit (2PC) [2PCREF] is supported by the protocol to achieve the transactional operations utilizing Config messages (Section 7.6.1). The COMMIT and TRCOMP operations in conjunction with the AT and the TP flags in the common header (Section 6.1) are provided for 2PC- based transactional operations spanning multiple messages. The AT flag, when set, indicates that this message belongs to an Atomic Transaction. All messages for a transaction operation MUST have the AT flag set. If not set, it means that the message is a stand-alone message and does not participate in any transaction operation that spans multiple messages. The TP flag indicates the Transaction Phase to which this message belongs. There are 4 possible phases for a transactional operation known as: SOT (Start of Transaction) MOT (Middle of Transaction) EOT (End of Transaction) ABT (Abort) The COMMIT operation is used by the CE to signal to the FE(s) to commit a transaction. When used with an ABT TP flag, the COMMIT operation signals the FE(s) to roll back (i.e., un-COMMIT) a previously committed transaction. The TRCOMP operation is a small addition to the classical 2PC approach. TRCOMP is sent by the CE to signal to the FE(s) that the transaction they have COMMITed is now over. This allows the FE(s) an opportunity to clear state they may have kept around to perform a roll back (if it became necessary). A transaction operation is started with a message in which the TP flag is set to Start of Transaction (SOT). Multi-part messages, after the first one, are indicated by the Middle of Transaction (MOT) flag. All messages from the CE MUST set the AlwaysACK flag (Section 6) to solicit responses from the FE(s). Before the CE issues a commit (described further below), the FE MUST only validate that the operation can be executed but not execute it.
Any failure notified by an FE causes the CE to abort the
transaction on all FEs involved in the transaction. This is
achieved by sending a Config message with an ABT flag and a COMMIT
operation.
If there are no failures by any participating FE, the transaction
commitment phase is signaled from the CE to the FE by an End of
Transaction (EOT) configuration message with a COMMIT operation.
The FE MUST respond to the CE's EOT message. There are two possible
failure scenarios in which the CE MUST abort the transaction (as
described above):
a. If any participating FE responds with a failure message in
relation to the transaction.
b. If no response is received from a participating FE within a
specified timeout.
If all participating FEs respond with a success indicator within the
expected time, then the CE MUST issue a TRCOMP operation to all
participating FEs. An FE MUST NOT respond to a TRCOMP.
Note that a transactional operation is generically atomic; therefore,
it requires that the execution modes of all messages in a transaction
operation should always be kept the same and be set to 'execute-all-
or-none'. If the EM flag is set to other execution modes, it will
result in a transaction failure.
As noted above, a transaction may span multiple messages. It is up
to the CE to keep track of the different outstanding messages making
up a transaction. As an example, the correlator field could be used
to mark transactions and a sequence field to label the different
messages within the same atomic transaction, but this is out of scope
and up to implementations.
4.3.1.2.3. Recovery
Any of the participating FEs or the CE or the associations between
them may fail after the EOT Response message has been sent by the FE
but before the CE has received all the responses, e.g., if the EOT
response never reaches the CE.
In this protocol revision, as indicated in Section 4.2.2.3, an FE
losing an association would be required to get entirely new state
from the newly associated CE upon a re-association. Although this
approach is simplistic and provides likeliness of losing data path
traffic, it is a design choice to avoid the additional complexity of managing graceful restarts. The HA mechanisms (Section 8) are provided to allow for a continuous operation in case of FE failures. Flexibility is provided on how to react when an FE loses association. This is dictated by the CE failover policy (refer to Section 8 and Section 7.3).4.3.1.2.4. Transaction Messaging Example
This section illustrates an example of how a successful two-phase commit between a CE and an FE would look in the simple case. FE PL CE PL | | | (1) Config, SOT,AT, EM=All-or-None, OP= SET/DEL,etc | |<-----------------------------------------------------| | | | (2) ACKnowledge | |----------------------------------------------------->| | | | (3) Config, MOT,AT, EM=All-or-None, OP= SET/DEL,etc | |<-----------------------------------------------------| | | | (4) ACKnowledge | |----------------------------------------------------->| | | | (5) Config, MOT,AT, EM=All-or-None, OP= SET/DEL,etc | |<-----------------------------------------------------| | | | (6) ACKnowledge | |----------------------------------------------------->| . . . . . . . . | | | (N) Config, EOT,AT, EM=All-or-None, OP= COMMIT | |<-----------------------------------------------------| | | | (N+1)Config-response, ACKnowledge, OP=COMMIT-RESPONSE| |----------------------------------------------------->| | | | (N+2) Config, OP=TRCOMP | |<-----------------------------------------------------| Figure 7: Example of a Two-Phase Commit
For the scenario illustrated above:
o In step 1, the CE issues a Config message with an operation of
choice like a DEL or SET. The transaction flags are set to
indicate a Start of Transaction (SOT), Atomic Transaction (AT),
and execute-all-or-none.
o The FE validates that it can execute the request successfully and
then issues an acknowledgment back to the CE in step 2.
o In step 3, the same sort of construct as in step 1 is repeated by
the CE with the transaction flag changed to Middle of Transaction
(MOT).
o The FE validates that it can execute the request successfully and
then issues an acknowledgment back to the CE in step 4.
o The CE-FE exchange continues in the same manner until all the
operations and their parameters are transferred to the FE. This
happens in step (N-1).
o In step N, the CE issues a commit. A commit is a Config message
with an operation of type COMMIT. The transaction flag is set to
End of Transaction (EOT). Essentially, this is an "empty" message
asking the FE to execute all the operations it has gathered since
the beginning of the transaction (message #1).
o The FE at this point executes the full transaction. It then
issues an acknowledgment back to the CE in step (N+1) that
contains a COMMIT-RESPONSE.
o The CE in this case has the simple task of issuing a TRCOMP
operation to the FE in step (N+2).
4.3.2. Scalability
It is desirable that the PL not become the bottleneck when larger
bandwidth pipes become available. To pick a hypothetical example in
today's terms, if a 100-Gbps pipe is available and there is
sufficient work, then the PL should be able to take advantage of this
and use all of the 100-Gbps pipe. Two mechanisms have been provided
to achieve this. The first one is batching and the second one is a
command pipeline.
Batching is the ability to send multiple commands (such as Config) in one Protocol Data Unit (PDU). The size of the batch will be affected by, among other things, the path MTU. The commands may be part of the same transaction or may be part of unrelated transactions that are independent of each other. Command pipelining allows for pipelining of independent transactions that do not affect each other. Each independent transaction could consist of one or more batches.4.3.2.1. Batching
There are several batching levels at different protocol hierarchies. o Multiple PL PDUs can be aggregated under one TML message. o Multiple LFB classes and instances (as indicated in the LFB selector) can be addressed within one PL PDU. o Multiple operations can be addressed to a single LFB class and instance.4.3.2.2. Command Pipelining
The protocol allows any number of messages to be issued by the CE before the corresponding acknowledgments (if requested) have been returned by the FE. Hence, pipelining is inherently supported by the protocol. Matching responses with requests messages can be done using the correlator field in the message header.4.3.3. Heartbeat Mechanism
Heartbeats (HBs) between FEs and CEs are traffic sensitive. An HB is sent only if no PL traffic is sent between the CE and FE within a configured interval. This has the effect of reducing the amount of HB traffic in the case of busy PL periods. An HB can be sourced by either the CE or FE. When sourced by the CE, a response can be requested (similar to the ICMP ping protocol). The FE can only generate HBs in the case of being configured to do so by the CE. Refer to Section 7.3.1 and Section 7.10 for details.
4.3.4. FE Object and FE Protocol LFBs
All PL messages operate on LFB constructs, as this provides more flexibility for future enhancements. This means that maintenance and configurability of FEs, NE, and the ForCES protocol itself MUST be expressed in terms of this LFB architecture. For this reason, special LFBs are created to accommodate this need. In addition, this shows how the ForCES protocol itself can be controlled by the very same type of structures (LFBs) it uses to control functions such as IP forwarding, filtering, etc. To achieve this, the following specialized LFBs are introduced: o FE Protocol LFB, which is used to control the ForCES protocol. o FE Object LFB, which is used to control components relative to the FE itself. Such components include FEState [RFC5812], vendor, etc. These LFBs are detailed in Section 7.3.4.4. Protocol Scenarios
This section provides a very high level description of sample message sequences between a CE and an FE. For protocol message encoding refer to Section 6.1, and for the semantics of the protocol refer to Section 4.3.4.4.1. Association Setup State
The associations among CEs and FEs are initiated via the Association Setup message from the FE. If a Setup Request is granted by the CE, a successful Setup Response message is sent to the FE. If CEs and FEs are operating in an insecure environment, then the security associations have to be established between them before any association messages can be exchanged. The TML MUST take care of establishing any security associations. This is typically followed by capability query, topology query, etc. When the FE is ready to start processing the data path, it sets the FEO FEState component to OperEnable (refer to [RFC5812] for details) and reports it to the CE as such when it is first queried. Typically, the FE is expected to be ready to process the data path before it associates, but there may be rare cases where it needs time do some pre-processing -- in such a case, the FE will start in the OperDisable state and when it is ready will transition to the OperEnable state. An example of an FE starting in OperDisable then
transitioning to OperEnable is illustrated in Figure 8. The CE could at any time also disable the FE's data path operations by setting the FEState to AdminDisable. The FE MUST NOT process packets during this state unless the CE sets the state back to OperEnable. These sequences of messages are illustrated in Figure 8 below. FE PL CE PL | | | Asso Setup Req | |---------------------->| | | | Asso Setup Resp | |<----------------------| | | | LFBx Query capability | |<----------------------| | | | LFBx Query Resp | |---------------------->| | | | FEO Query (Topology) | |<----------------------| | | | FEO Query Resp | |---------------------->| | | | FEO OperEnable Event | |---------------------->| | | | Config FEO Adminup | |<----------------------| | | | FEO Config-Resp | |---------------------->| | | Figure 8: Message Exchange between CE and FE to Establish an NE Association On successful completion of this state, the FE joins the NE.
4.4.2. Association Established State or Steady State
In this state, the FE is continuously updated or queried. The FE may also send asynchronous event notifications to the CE, synchronous Heartbeat messages, or Packet Redirect message to the CE. This continues until a termination (or deactivation) is initiated by either the CE or FE. Figure 9 below, helps illustrate this state.
FE PL CE PL | | | Heartbeat | |<---------------------------->| | | | Heartbeat | |----------------------------->| | | | Config-set LFBy (Event sub.) | |<-----------------------------| | | | Config Resp LFBy | |----------------------------->| | | | Config-set LFBx Component | |<-----------------------------| | | | Config Resp LFBx | |----------------------------->| | | |Config-Query LFBz (Stats) | |<--------------------------- -| | | | Query Resp LFBz | |----------------------------->| | | | FE Event Report | |----------------------------->| | | | Config-Del LFBx Component | |<-----------------------------| | | | Config Resp LFBx | |----------------------------->| | | | Packet Redirect LFBx | |----------------------------->| | | | Heartbeat | |<-----------------------------| . . . . | | Figure 9: Message Exchange between CE and FE during Steady-State Communication
Note that the sequence of messages shown in the figure serve only as examples and the message exchange sequences could be different from what is shown in the figure. Also, note that the protocol scenarios described in this section do not include all the different message exchanges that would take place during failover. That is described in the HA section (Section 8).
(page 31 continued on part 3)
