RFC 5046
Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)
Network Working Group M. Ko Request for Comments: 5046 IBM Corporation Category: Standards Track M. Chadalapaka Hewlett-Packard Company J. Hufferd Brocade, Inc. U. Elzur H. Shah P. Thaler Broadcom Corporation October 2007 Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA) Status of This Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.Abstract
Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA) provides the RDMA data transfer capability to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol, such as the iWARP protocol suite. An RDMA-Capable Protocol provides RDMA Read and Write services, which enable data to be transferred directly into SCSI I/O Buffers without intermediate data copies. This document describes the extensions to the iSCSI protocol to support RDMA services as provided by an RDMA-Capable Protocol, such as the iWARP protocol suite.
Table of Contents
1. Introduction ....................................................5 1.1. Motivation .................................................5 1.2. Architectural Goals ........................................6 1.3. Protocol Overview ..........................................7 1.4. RDMA Services and iSER .....................................8 1.4.1. STag ................................................8 1.4.2. Send ................................................9 1.4.3. RDMA Write ..........................................9 1.4.4. RDMA Read ...........................................9 1.5. SCSI Read Overview ........................................10 1.6. SCSI Write Overview .......................................10 1.7. iSCSI/iSER Layering .......................................10 2. Definitions and Acronyms .......................................11 2.1. Definitions ...............................................11 2.2. Acronyms ..................................................17 2.3. Conventions ...............................................19 3. Upper Layer Interface Requirements .............................19 3.1. Operational Primitives Offered by iSER ....................20 3.1.1. Send_Control .......................................20 3.1.2. Put_Data ...........................................20 3.1.3. Get_Data ...........................................21 3.1.4. Allocate_Connection_Resources ......................21 3.1.5. Deallocate_Connection_Resources ....................22 3.1.6. Enable_Datamover ...................................22 3.1.7. Connection_Terminate ...............................22 3.1.8. Notice_Key_Values ..................................23 3.1.9. Deallocate_Task_Resources ..........................23 3.2. Operational Primitives Used by iSER .......................23 3.2.1. Control_Notify .....................................24 3.2.2. Data_Completion_Notify .............................24 3.2.3. Data_ACK_Notify ....................................24 3.2.4. Connection_Terminate_Notify ........................25 3.3. iSCSI Protocol Usage Requirements .........................25 4. Lower Layer Interface Requirements .............................26 4.1. Interactions with the RCaP Layer ..........................26 4.2. Interactions with the Transport Layer .....................27 5. Connection Setup and Termination ...............................27 5.1. iSCSI/iSER Connection Setup ...............................27 5.1.1. Initiator Behavior .................................29 5.1.2. Target Behavior ....................................30 5.1.3. iSER Hello Exchange ................................32 5.2. iSCSI/iSER Connection Termination .........................33 5.2.1. Normal Connection Termination at the Initiator .....33 5.2.2. Normal Connection Termination at the Target ........34 5.2.3. Termination without Logout Request/Response PDUs ...34
6. Login/Text Operational Keys ....................................35 6.1. HeaderDigest and DataDigest ...............................35 6.2. MaxRecvDataSegmentLength ..................................36 6.3. RDMAExtensions ............................................36 6.4. TargetRecvDataSegmentLength ...............................37 6.5. InitiatorRecvDataSegmentLength ............................38 6.6. OFMarker and IFMarker .....................................38 6.7. MaxOutstandingUnexpectedPDUs ..............................38 7. iSCSI PDU Considerations .......................................39 7.1. iSCSI Data-Type PDU .......................................39 7.2. iSCSI Control-Type PDU ....................................40 7.3. iSCSI PDUs ................................................40 7.3.1. SCSI Command .......................................40 7.3.2. SCSI Response ......................................42 7.3.3. Task Management Function Request/Response ..........44 7.3.4. SCSI Data-Out ......................................45 7.3.5. SCSI Data-In .......................................46 7.3.6. Ready to Transfer (R2T) ............................48 7.3.7. Asynchronous Message ...............................50 7.3.8. Text Request and Text Response .....................50 7.3.9. Login Request and Login Response ...................50 7.3.10. Logout Request and Logout Response ................51 7.3.11. SNACK Request .....................................51 7.3.12. Reject ............................................51 7.3.13. NOP-Out and NOP-In ................................51 8. Flow Control and STag Management ...............................52 8.1. Flow Control for RDMA Send Message Types ..................52 8.1.1. Flow Control for Control-Type PDUs from the Initiator ..........................................52 8.1.2. Flow Control for Control-Type PDUs from the Target .............................................55 8.2. Flow Control for RDMA Read Resources ......................56 8.3. STag Management ...........................................56 8.3.1. Allocation of STags ................................57 8.3.2. Invalidation of STags ..............................57 9. iSER Control and Data Transfer .................................58 9.1. iSER Header Format ........................................58 9.2. iSER Header Format for the iSCSI Control-Type PDU .........59 9.3. iSER Header Format for the iSER Hello Message .............60 9.4. iSER Header Format for the iSER HelloReply Message ........61 9.5. SCSI Data Transfer Operations .............................62 9.5.1. SCSI Write Operation ...............................62 9.5.2. SCSI Read Operation ................................63 9.5.3. Bidirectional Operation ............................64 10. iSER Error Handling and Recovery ..............................64 10.1. Error Handling ...........................................64 10.1.1. Errors in the Transport Layer .....................64 10.1.2. Errors in the RCaP Layer ..........................65
10.1.3. Errors in the iSER Layer ..........................66
10.1.4. Errors in the iSCSI Layer .........................67
10.2. Error Recovery ...........................................69
10.2.1. PDU Recovery ......................................69
10.2.2. Connection Recovery ...............................70
11. Security Considerations .......................................71
12. References ....................................................71
12.1. Normative References .....................................71
12.2. Informative References ...................................72
Appendix A. iWARP Message Format for iSER .........................73
A.1. iWARP Message Format for iSER Hello Message ...............73
A.2. iWARP Message Format for iSER HelloReply Message ..........74
A.3. iWARP Message Format for SCSI Read Command PDU ............75
A.4. iWARP Message Format for SCSI Read Data ...................76
A.5. iWARP Message Format for SCSI Write Command PDU ...........77
A.6. iWARP Message Format for RDMA Read Request ................78
A.7. iWARP Message Format for Solicited SCSI Write Data ........79
A.8. iWARP Message Format for SCSI Response PDU ................80
Appendix B. Architectural Discussion of iSER over InfiniBand ......81
B.1. The Host Side of the iSCSI and iSER Connections
in InfiniBand .............................................81
B.2. The Storage Side of the iSCSI and iSER Mixed
Network Environment .......................................82
B.3. Discovery Processes for an InfiniBand Host ................82
B.4. IBTA Connection Specifications ............................83
Acknowledgments ...................................................83
Table of Figures
Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase ....11
Figure 2. iSER Header Format ......................................58
Figure 3. iSER Header Format for iSCSI Control-Type PDU ...........59
Figure 4. iSER Header Format for iSER Hello Message ...............60
Figure 5. iSER Header Format for iSER HelloReply Message ..........61
Figure 6. SendSE Message containing an iSER Hello Message .........72
Figure 7. SendSE Message containing an iSER HelloReply Message ....74
Figure 8. SendSE Message containing a SCSI Read Command PDU .......75
Figure 9. RDMA Write Message containing SCSI Read Data ............76
Figure 10. SendSE Message containing a SCSI Write Command PDU .....77
Figure 11. RDMA Read Request Message ..............................78
Figure 12. RDMA Read Response Message containing SCSI Write Data ..79
Figure 13. SendInvSE Message containing SCSI Response PDU .........80
Figure 14. iSCSI and iSER on IB ...................................81
Figure 15. Storage Controller with TCP, iWARP, and IB Connections .82
1. Introduction
1.1. Motivation
The iSCSI protocol [RFC3720] is a mapping of the SCSI Architecture Model (see [SAM2]) over the TCP protocol. SCSI commands are carried by iSCSI requests, and SCSI responses and status are carried by iSCSI responses. Other iSCSI protocol exchanges and SCSI data are also transported in iSCSI Protocol Data Units (PDUs). Out-of-order TCP segments in the Traditional iSCSI model have to be stored and reassembled before the iSCSI protocol layer within an end node can place the data in the iSCSI buffers. This reassembly is required because not every TCP segment is likely to contain an iSCSI header to enable its placement, and TCP itself does not have a built-in mechanism for signaling Upper Level Protocol (ULP) message boundaries to aid placement of out-of-order segments. This TCP reassembly at high network speeds is quite counter-productive for the following reasons: wasted memory bandwidth in data copying, the need for reassembly memory, wasted CPU cycles in data copying, and the general store-and-forward latency from an application perspective. TCP reassembly was recognized as a serious issue in [RFC3720], and the notion of a "sync and steering layer" was introduced that is optional to implement and use. One specific sync and steering mechanism, called "markers", was defined in [RFC3720], which provides an application-level way of framing iSCSI Protocol Data Units (PDUs) within the TCP data stream even when the TCP segments are not yet reassembled to be in-order. With these defined techniques in [RFC3720], a Network Interface Controller customized for iSCSI (SNIC) could offload the TCP/IP processing and support direct data placement, but most iSCSI implementations do not support iSCSI "markers", making SNIC marker- based direct data placement unusable in practice. The iWARP protocol stack provides direct data placement functionality that is usable in practice. In addition, there is interest in using iSCSI with other Remote Direct Memory Access (RDMA) protocol stacks that support direct data placement, such as the one provided by InfiniBand. The generic term RDMA-Capable Protocol (RCaP) is used to refer to the RDMA functionality provided by such protocol stacks. With the availability of RDMA-Capable Controllers within a host system, which does not have SNICs, it is appropriate for iSCSI to be able to exploit the direct data placement function of the RDMA- Capable Controller like other applications.
iSCSI Extensions for RDMA (iSER) is designed precisely to take advantage of generic RDMA technologies -- iSER's goal is to permit iSCSI to employ direct data placement and RDMA capabilities using a generic RDMA-Capable Controller. In summary, the iSCSI/iSER protocol stack is designed to enable scaling to high speeds by relying on a generic data placement process and RDMA technologies and products, which enable direct data placement of both in-order and out-of-order data. This document describes iSER as a protocol extension to iSCSI, both for convenience of description and because it is true in a very strict protocol sense. However, note that iSER is in reality extending the connectivity of the iSCSI protocol defined in [RFC3720], and the name iSER reflects this reality. When the iSCSI protocol as defined in [RFC3720] (i.e., without the iSER enhancements) is intended in the rest of the document, the term "Traditional iSCSI" is used to make the intention clear.1.2. Architectural Goals
This section summarizes the architectural goals that guided the design of iSER. 1. Provide an RDMA data transfer model for iSCSI that enables direct in-order or out-of-order data placement of SCSI data into pre- allocated SCSI buffers while maintaining in-order data delivery. 2. Not require any major changes to the SCSI Architecture Model [SAM2] and SCSI command set standards. 3. Utilize existing iSCSI infrastructure (sometimes referred to as "iSCSI ecosystem") including but not limited to MIB, bootstrapping, negotiation, naming and discovery, and security. 4. Require a session to operate in the Traditional iSCSI data transfer mode if iSER is not supported by either the initiator or the target (i.e., not require iSCSI Full Feature Phase interoperability between an end node operating in Traditional iSCSI mode, and an end node operating in iSER-assisted mode). 5. Allow initiator and target implementations to utilize generic RDMA-Capable Controllers such as RDMA-enabled Network Interface Controllers (RNICs), or to implement iSCSI and iSER in software (not require iSCSI- or iSER-specific assists in the RCaP implementation or RDMA-Capable Controller).
6. Require full and only generic RCaP functionality at both the
initiator and the target.
7. Implement a lightweight Datamover protocol for iSCSI with minimal
state maintenance.
1.3. Protocol Overview
Consistent with the architectural goals stated in Section 2.2, the
iSER protocol does not require changes in the iSCSI ecosystem or any
related SCSI specifications. The iSER protocol defines the mapping
of iSCSI PDUs to RCaP Messages in such a way that it is entirely
feasible to realize iSCSI/iSER implementations that are based on
generic RDMA-Capable Controllers. The iSER protocol layer requires
minimal state maintenance to assist an iSCSI Full Feature Phase
connection, besides being oblivious to the notion of an iSCSI
session. The crucial protocol aspects of iSER may be summarized
thus:
1. iSER-assisted mode is negotiated during the iSCSI login for each
session, and an entire iSCSI session can only operate in one mode
(i.e., a connection in a session cannot operate in iSER-assisted
mode if a different connection of the same session is already in
Full Feature Phase in the Traditional iSCSI mode).
2. Once in iSER-assisted mode, all iSCSI interactions on that
connection use RCaP Messages.
3. A Send Message Type is used for carrying an iSCSI control-type PDU
preceded by an iSER header. See Section 7.2 for more details on
iSCSI control-type PDUs.
4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages are
used for carrying control and all data information associated with
the iSCSI data-type PDUs. See Section 7.1 for more details on
iSCSI data-type PDUs.
5. Target drives all data transfer (with the exception of iSCSI
unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA
Read Requests and RDMA Writes, respectively.
6. RCaP is responsible for ensuring data integrity. (For example,
iWARP includes a CRC-enhanced framing layer called Marker PDU
Aligned Framing for TCP (MPA) on top of TCP; and for InfiniBand,
the CRCs are included in the Reliable Connection mode). For this
reason, iSCSI header and data digests are negotiated to "None" for
iSCSI/iSER sessions.
7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully supported by iSER. (However, see Section 7.3.11 on the handling of SNACK Request PDUs.) 8. iSER requires no changes to iSCSI authentication, security, and text mode negotiation mechanisms. Note that Traditional iSCSI implementations may have to be adapted to employ iSER. It is expected that the adaptation when required is likely to be centered around the upper layer interface requirements of iSER (Section 3).1.4. RDMA Services and iSER
iSER is designed to work with software and/or hardware protocol stacks providing the protocol services defined in RCaP documents such as [RDMAP], [IB], etc. The following subsections describe the key protocol elements of RCaP services that iSER relies on.1.4.1. STag
A Steering Tag (STag) is the identifier of an I/O Buffer unique to an RDMA-Capable Controller that the iSER layer Advertises to the remote iSCSI/iSER node in order to complete a SCSI I/O. In iSER, Advertisement is the act of informing the target by the initiator that an I/O Buffer is available at the initiator for RDMA Read or RDMA Write access by the target. The initiator Advertises the I/O Buffer by including the STag in the header of an iSER Message containing the SCSI Command PDU to the target. The base Tagged Offset is not explicitly specified, but the target must always assume it as zero. The buffer length is as specified in the SCSI Command PDU. The iSER layer at the initiator Advertises the STag for the I/O Buffer of each SCSI I/O to the iSER layer at the target in the iSER header of the Send with Solicited Event (SendSE) Message containing the SCSI Command PDU, unless the I/O can be completely satisfied by unsolicited data alone. The iSER layer at the target provides the STag for the I/O Buffer that is the Data Sink of an RDMA Read Operation (Section 2.4.4) to the RCaP layer on the initiator node -- i.e., this is completely transparent to the iSER layer at the initiator. The iSER protocol is defined so that the Advertised STag is automatically invalidated upon a normal completion of the associated task. This automatic invalidation is realized via the Send with
Solicited Event and Invalidate (SendInvSE) Message carrying the SCSI Response PDU. There are two exceptions to this automatic invalidation -- bidirectional commands, and abnormal completion of a command. The iSER layer at the initiator is required to explicitly invalidate the STag in these cases, in addition to sanity checking the automatic invalidation even when that does happen.1.4.2. Send
Send is the RDMA Operation that is not addressed to an Advertised buffer by the sending side, and thus uses Untagged buffers on the receiving side. The iSER layer at the initiator uses the Send Operation to transmit any iSCSI control-type PDU to the target. As an example, the initiator uses Send Operations to transfer iSER Messages containing SCSI Command PDUs to the iSER layer at the target. An iSER layer at the target uses the Send Operation to transmit any iSCSI control-type PDU to the initiator. As an example, the target uses Send Operations to transfer iSER Messages containing SCSI Response PDUs to the iSER layer at the initiator.1.4.3. RDMA Write
RDMA Write is the RDMA Operation that is used to place data into an Advertised buffer on the receiving side. The sending side addresses the Message using an STag and a Tagged Offset that are valid on the Data Sink. The iSER layer at the target uses the RDMA Write Operation to transfer the contents of a local I/O Buffer to an Advertised I/O Buffer at the initiator. The iSER layer at the target uses the RDMA Write to transfer whole or part of the data required to complete a SCSI read command. The iSER layer at the initiator does not employ RDMA Writes.1.4.4. RDMA Read
RDMA Read is the RDMA Operation that is used to retrieve data from an Advertised buffer on a remote node. The sending side of the RDMA Read Request addresses the Message using an STag and a Tagged Offset that are valid on the Data Source in addition to providing a valid local STag and Tagged Offset that identify the Data Sink. The iSER layer at the target uses the RDMA Read Operation to transfer the contents of an Advertised I/O Buffer at the initiator to a local
I/O Buffer at the target. The iSER layer at the target uses the RDMA Read to fetch whole or part of the data required to complete a SCSI write command. The iSER layer at the initiator does not employ RDMA Reads.1.5. SCSI Read Overview
The iSER layer at the initiator receives the SCSI Command PDU from the iSCSI layer. The iSER layer at the initiator generates an STag for the I/O Buffer of the SCSI Read and Advertises the buffer by including the STag as part of the iSER header for the PDU. The iSER Message is transferred to the target using a SendSE Message. The iSER layer at the target uses one or more RDMA Writes to transfer the data required to complete the SCSI Read. The iSER layer at the target uses a SendInvSE Message to transfer the SCSI Response PDU back to the iSER layer at the initiator. The iSER layer at the initiator notifies the iSCSI layer of the availability of the SCSI Response PDU.1.6. SCSI Write Overview
The iSER layer at the initiator receives the SCSI Command PDU from the iSCSI layer. If solicited data transfer is involved, the iSER layer at the initiator generates an STag for the I/O Buffer of the SCSI Write and Advertises the buffer by including the STag as part of the iSER header for the PDU. The iSER Message is transferred to the target using a SendSE Message. The iSER layer at the initiator may optionally send one or more non- immediate unsolicited data PDUs to the target using Send Message Types. If solicited data transfer is involved, the iSER layer at the target uses one or more RDMA Reads to transfer the data required to complete the SCSI Write. The iSER layer at the target uses a SendInvSE Message to transfer the SCSI Response PDU back to the iSER layer at the initiator. The iSER layer at the initiator notifies the iSCSI layer of the availability of the SCSI Response PDU.1.7. iSCSI/iSER Layering
iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer and the RCaP layer. Note that the RCaP layer may be composed of one
or more distinct protocol layers depending on the specifics of the RCaP. Figure 1 shows an example of the relationship between SCSI, iSCSI, iSER, and the different RCaP layers. For TCP, the RCaP is iWARP. For InfiniBand, the RCaP is the Reliable Connected Transport Service. Note that the iSCSI layer as described here supports the RDMA Extensions as used in iSER. +-------------------------------------+ | SCSI | +-------------------------------------+ | iSCSI | DI ------> +-------------------------------------+ | iSER | +---------+--------------+------------+ | RDMAP | | | +---------+ InfiniBand | | | DDP | Reliable | Other | +---------+ Connected | RDMA- | | MPA | Transport | Capable | +---------+ Service | Protocol | | TCP | | | +---------+--------------+------------+ | | InfiniBand | Other | | IP | Network | Network | | | Layer | Layer | +---------+--------------+------------+ Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase2. Definitions and Acronyms
2.1. Definitions
Advertisement (Advertised, Advertise, Advertisements, Advertises) - The act of informing a remote iSER layer that a local node's buffer is available to it. A Node makes a buffer available for incoming RDMA Read Request Message or incoming RDMA Write Message access by informing the remote iSER layer of the Tagged Buffer identifiers (STag, TO, and buffer length). Note that this Advertisement of Tagged Buffer information is the responsibility of the iSER layer on either end and is not defined by the RDMA- Capable Protocol. A typical method would be for the iSER layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message destined for the remote iSER layer. Completion (Completed, Complete, Completes) - Completion is defined as the process by the RDMA-Capable Protocol layer to inform the
iSER layer, that a particular RDMA Operation has performed all
functions specified for the RDMA Operation.
Connection - A connection is a logical circuit between the initiator
and the target, e.g., a TCP connection. Communication between the
initiator and the target occurs over one or more connections. The
connections carry control messages, SCSI commands, parameters, and
data within iSCSI Protocol Data Units (iSCSI PDUs).
Connection Handle - An information element that identifies the
particular iSCSI connection and is unique for a given iSCSI-iSER
pair. Every invocation of an Operational Primitive is qualified
with the Connection Handle.
Data Sink - The peer receiving a data payload. Note that the Data
Sink can be required to both send and receive RCaP Messages to
transfer a data payload.
Data Source - The peer sending a data payload. Note that the Data
Source can be required to both send and receive RCaP Messages to
transfer a data payload.
Datamover Interface (DI) - The interface between the iSCSI layer and
the Datamover layer as described in [DA].
Datamover Layer - A layer that is directly below the iSCSI layer and
above the underlying transport layers. This layer exposes and
uses a set of transport independent Operational Primitives for the
communication between the iSCSI layer and itself. The Datamover
layer, operating in conjunction with the transport layers, moves
the control and data information on the iSCSI connection. In this
specification, the iSER layer is the Datamover layer.
Datamover Protocol - A Datamover protocol is the wire-protocol that
is defined to realize the Datamover layer functionality. In this
specification, the iSER protocol is the Datamover protocol.
Event - An indication provided by the RDMA-Capable Protocol layer to
the iSER layer to indicate a Completion or other condition
requiring immediate attention.
Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
outstanding RDMA Read Requests that the RDMA-Capable Controller
can handle on a particular RCaP Stream at the Data Source. For
some RDMA-Capable Protocol layers, the term "IRD" may be known by
a different name. For example, for InfiniBand, the equivalent for
IRD is the Responder Resources.
Invalidate STag - A mechanism used to prevent the Remote Peer from
reusing a previous explicitly Advertised STag, until the iSER
layer at the local node makes it available through a subsequent
explicit Advertisement.
I/O Buffer - A buffer that is used in a SCSI Read or Write operation
so SCSI data may be sent from or received into that buffer.
iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of
the SCSI Architecture Model of SAM-2 over TCP.
iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data-
type PDU and also not a SCSI Data-out PDU carrying solicited data
is defined as an iSCSI control-type PDU. Specifically, it is to
be noted that SCSI Data-out PDUs for unsolicited data are defined
as iSCSI control-type PDUs.
iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
PDU that causes data transfer, transparent to the remote iSCSI
layer, to take place between the peer iSCSI nodes on a Full
Feature Phase iSCSI connection. An iSCSI data-type PDU, when
requested for transmission by the sender iSCSI layer, results in
the associated data transfer without the participation of the
remote iSCSI layer, i.e. the PDU itself is not delivered as-is to
the remote iSCSI layer. The following iSCSI PDUs constitute the
set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU.
iSCSI Layer - A layer in the protocol stack implementation within an
end node that implements the iSCSI protocol and interfaces with
the iSER layer via the Datamover Interface.
iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the
initiator and the iSCSI layer at the target divide their
communications into messages. The term "iSCSI protocol data unit"
(iSCSI PDU) is used for these messages.
iSCSI/iSER Connection - An iSER-assisted iSCSI connection.
iSCSI/iSER Session - An iSER-assisted iSCSI session.
iSCSI-iSER Pair - The iSCSI layer and the underlying iSER layer.
iSER - iSCSI Extensions for RDMA, the protocol defined in this
document.
iSER-assisted - A term generally used to describe the operation of
iSCSI when the iSER functionality is also enabled below the iSCSI
layer for the specific iSCSI/iSER connection in question.
iSER-IRD - This variable represents the maximum number of incoming
outstanding RDMA Read Requests that the iSER layer at the
initiator declares on a particular RCaP Stream.
iSER-ORD - This variable represents the maximum number of outstanding
RDMA Read Requests that the iSER layer can initiate on a
particular RCaP Stream. This variable is maintained only by the
iSER layer at the target.
iSER Layer - The layer that implements the iSCSI Extensions for RDMA
(iSER) protocol.
iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
[MPA] when layered above [TCP]. [RDMAP] and [DDP] may be layered
above SCTP or other transport protocols.
Local Mapping - A task state record maintained by the iSER layer that
associates the Initiator Task Tag to the local STag(s). The
specifics of the record structure are implementation dependent.
Local Peer - The implementation of the RDMA-Capable Protocol on the
local end of the connection. Used to refer to the local entity
when describing protocol exchanges or other interactions between
two Nodes.
Node - A computing device attached to one or more links of a network.
A Node in this context does not refer to a specific application or
protocol instantiation running on the computer. A Node may
consist of one or more RDMA-Capable Controllers installed in a
host computer.
Operational Primitive - An Operational Primitive is an abstract
functional interface procedure that requests that another layer
perform a specific action on the requestor's behalf or notifies
the other layer of some event. The Datamover Interface between an
iSCSI layer and a Datamover layer within an iSCSI end node uses a
set of Operational Primitives to define the functional interface
between the two layers. Note that not every invocation of an
Operational Primitive may elicit a response from the requested
layer. A full discussion of the Operational Primitive types and
request-response semantics available to iSCSI and iSER can be
found in [DA].
Outbound RDMA Read Queue Depth (ORD) - The maximum number of
outstanding RDMA Read Requests that the RDMA-Capable Controller
can initiate on a particular RCaP Stream at the Data Sink. For
some RDMA-Capable Protocol layer, the term "ORD" may be known by a
different name. For example, for InfiniBand, the equivalent for
ORD is the Initiator Depth.
Phase-Collapse - Refers to the optimization in iSCSI where the SCSI
status is transferred along with the final SCSI Data-in PDU from a
target. See Section 3.2 in [RFC3720].
RCaP Message - One or more packets of the network layer comprising a
single RDMA Operation or a part of an RDMA Read Operation of the
RDMA-Capable Protocol. For iWARP, an RCaP Message is known as an
RDMAP Message.
RCaP Stream - A single bidirectional association between the peer
RDMA-Capable Protocol layers on two Nodes over a single
transport-level stream. For iWARP, an RCaP Stream is known as an
RDMAP Stream, and the association is created when the connection
transitions to iSER-assisted mode following a successful Login
Phase during which iSER support is negotiated.
RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
provides a reliable RDMA transport functionality, e.g., iWARP,
InfiniBand, etc.
RDMA-Capable Controller - A network I/O adapter or embedded
controller with RDMA functionality. For example, for iWARP, this
could be an RNIC, and for InfiniBand, this could be a HCA (Host
Channel Adapter) or TCA (Target Channel Adapter).
RDMA-enabled Network Interface Controller (RNIC) - A network I/O
adapter or embedded controller with iWARP functionality.
RDMA Operation - A sequence of RCaP Messages, including control
Messages, to transfer data from a Data Source to a Data Sink. The
following RDMA Operations are defined - RDMA Write Operation, RDMA
Read Operation, Send Operation, Send with Invalidate Operation,
Send with Solicited Event Operation, Send with Solicited Event and
Invalidate Operation, and Terminate Operation.
RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations
to transfer ULP data between a Local Peer and the Remote Peer as
described in [RDMAP].
RDMA Read Operation - An RDMA Operation used by the Data Sink to
transfer the contents of a Data Source buffer from the Remote Peer
to a Data Sink buffer at the Local Peer. An RDMA Read operation
consists of a single RDMA Read Request Message and a single RDMA
Read Response Message.
RDMA Read Request - An RCaP Message used by the Data Sink to request
that the Data Source transfer the contents of a buffer. The RDMA
Read Request Message describes both the Data Source and the Data
Sink buffers.
RDMA Read Response - An RCaP Message used by the Data Source to
transfer the contents of a buffer to the Data Sink, in response to
an RDMA Read Request. The RDMA Read Response Message only
describes the Data Sink buffer.
RDMA Write Operation - An RDMA Operation used by the Data Source to
transfer the contents of a Data Source buffer from the Local Peer
to a Data Sink buffer at the Remote Peer. The RDMA Write Message
only describes the Data Sink buffer.
Remote Direct Memory Access (RDMA) - A method of accessing memory on
a remote system in which the local system specifies the remote
location of the data to be transferred. Employing an RDMA-
Capable Controller in the remote system allows the access to take
place without interrupting the processing of the CPU(s) on the
system.
Remote Mapping - A task state record maintained by the iSER layer
that associates the Initiator Task Tag to the Advertised STag(s).
The specifics of the record structure are implementation
dependent.
Remote Peer - The implementation of the RDMA-Capable Protocol on the
opposite end of the connection. Used to refer to the remote
entity when describing protocol exchanges or other interactions
between two Nodes.
SCSI Layer - This layer builds/receives SCSI CDBs (Command Descriptor
Blocks) and sends/receives them with the remaining command execute
[SAM2] parameters to/from the iSCSI layer.
Send - An RDMA Operation that transfers the contents of a Buffer from
the Local Peer to a Buffer at the Remote Peer.
Send Message Type - A Send Message, Send with Invalidate Message,
Send with Solicited Event Message, or Send with Solicited Event
and Invalidate Message.
SendInvSE Message - A Send with Solicited Event and Invalidate
Message.
SendSE Message - A Send with Solicited Event Message.
Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an
R2T PDU. The semantics for both types of sequence numbers are as
defined in [RFC3720].
Session, iSCSI Session - The group of connections that link an
initiator SCSI port with a target SCSI port form an iSCSI session
(equivalent to a SCSI I-T nexus). Connections can be added to and
removed from a session even while the I-T nexus is intact. Across
all connections within a session, an initiator sees one and the
same target.
Solicited Event (SE) - A facility by which an RDMA Operation sender
may cause an Event to be generated at the recipient, if the
recipient is configured to generate such an Event, when a Send
with Solicited Event or Send with Solicited Event and Invalidate
Message is received.
Steering Tag (STag) - An identifier of a Tagged Buffer on a Node
(Local or Remote) as defined in [RDMAP] and [DDP]. For other
RDMA-Capable Protocols, the Steering Tag may be known by different
names but will be herein referred to as STags. For example, for
InfiniBand, a Remote STag is known as an R-Key, and a local STag
is known as an L-Key, and both will be considered STags.
Tagged Buffer - A buffer that is explicitly Advertised to the iSER
layer at the remote node through the exchange of an STag, Tagged
Offset, and length.
Tagged Offset (TO) - The offset within a Tagged Buffer.
Traditional iSCSI - Refers to the iSCSI protocol as defined in
[RFC3720] (i.e. without the iSER enhancements).
Untagged Buffer - A buffer that is not explicitly Advertised to the
iSER layer at the remode node.
2.2. Acronyms
Acronym Definition
--------------------------------------------------------------
AHS Additional Header Segment
BHS Basic Header Segment
CO Connection Only
CRC Cyclic Redundancy Check
DDP Direct Data Placement Protocol
DI Datamover Interface
HCA Host Channel Adapter
IANA Internet Assigned Numbers Authority
IB InfiniBand
IETF Internet Engineering Task Force
I/O Input - Output
IO Initialize Only
IP Internet Protocol
IPoIB IP over InfiniBand
IPsec Internet Protocol Security
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
NOP No Operation
NSG Next Stage (during the iSCSI Login Phase)
OS Operating System
PDU Protocol Data Unit
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
RNIC RDMA-enabled Network Interface Controller
SAM2 SCSI Architecture Model - 2
SCSI Small Computer Systems Interface
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for data
STag Steering Tag
SW Session Wide
TCA Target Channel Adapter
TCP Transmission Control Protocol
TMF Task Management Function
TTT Target Transfer Tag
TO Tagged Offset
ULP Upper Level Protocol
2.3. Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
(page 19 continued on part 2)
