RFC 8435
Parallel NFS (pNFS) Flexible File Layout
Pages: 42
Proposed Standard
Proposed Standard
Part 2 of 2 – Pages 25 to 42
6. Striping via Sparse Mapping
While other layout types support both dense and sparse mapping of logical offsets to physical offsets within a file (see, for example, Section 13.4 of [RFC5661]), the flexible file layout type only supports a sparse mapping. With sparse mappings, the logical offset within a file (L) is also the physical offset on the storage device. As detailed in Section 13.4.4 of [RFC5661], this results in holes across each storage device that does not contain the current stripe index. L: logical offset within the file W: stripe width W = number of elements in ffm_data_servers S: number of bytes in a stripe S = W * ffl_stripe_unit N: stripe number N = L / S7. Recovering from Client I/O Errors
The pNFS client may encounter errors when directly accessing the storage devices. However, it is the responsibility of the metadata server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES layout type is used, the client MUST report the I/O errors to the server at LAYOUTRETURN time using the ff_ioerr4 structure (see Section 9.1.1). The metadata server analyzes the error and determines the required recovery operations such as recovering media failures or reconstructing missing data files. The metadata server MUST recall any outstanding layouts to allow it exclusive write access to the stripes being recovered and to prevent other clients from hitting the same error condition. In these cases, the server MUST complete recovery before handing out any new layouts to the affected byte ranges. Although the client implementation has the option to propagate a corresponding error to the application that initiated the I/O operation and drop any unwritten data, the client should attempt to retry the original I/O operation by either requesting a new layout or sending the I/O via regular NFSv4.1+ READ or WRITE operations to the metadata server. The client SHOULD attempt to retrieve a new layout
and retry the I/O operation using the storage device first and only retry the I/O operation via the metadata server if the error persists.8. Mirroring
The flexible file layout type has a simple model in place for the mirroring of the file data constrained by a layout segment. There is no assumption that each copy of the mirror is stored identically on the storage devices. For example, one device might employ compression or deduplication on the data. However, the over-the-wire transfer of the file contents MUST appear identical. Note, this is a constraint of the selected XDR representation in which each mirrored copy of the layout segment has the same striping pattern (see Figure 1). The metadata server is responsible for determining the number of mirrored copies and the location of each mirror. While the client may provide a hint to how many copies it wants (see Section 12), the metadata server can ignore that hint; in any event, the client has no means to dictate either the storage device (which also means the coupling and/or protocol levels to access the layout segments) or the location of said storage device. The updating of mirrored layout segments is done via client-side mirroring. With this approach, the client is responsible for making sure modifications are made on all copies of the layout segments it is informed of via the layout. If a layout segment is being resilvered to a storage device, that mirrored copy will not be in the layout. Thus, the metadata server MUST update that copy until the client is presented it in a layout. If the FF_FLAGS_WRITE_ONE_MIRROR is set in ffl_flags, the client need only update one of the mirrors (see Section 8.2). If the client is writing to the layout segments via the metadata server, then the metadata server MUST update all copies of the mirror. As seen in Section 8.3, during the resilvering, the layout is recalled, and the client has to make modifications via the metadata server.8.1. Selecting a Mirror
When the metadata server grants a layout to a client, it MAY let the client know how fast it expects each mirror to be once the request arrives at the storage devices via the ffds_efficiency member. While the algorithms to calculate that value are left to the metadata server implementations, factors that could contribute to that calculation include speed of the storage device, physical memory available to the device, operating system version, current load, etc.
However, what should not be involved in that calculation is a perceived network distance between the client and the storage device. The client is better situated for making that determination based on past interaction with the storage device over the different available network interfaces between the two; that is, the metadata server might not know about a transient outage between the client and storage device because it has no presence on the given subnet. As such, it is the client that decides which mirror to access for reading the file. The requirements for writing to mirrored layout segments are presented below.8.2. Writing to Mirrors
8.2.1. Single Storage Device Updates Mirrors
If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client only needs to update one of the copies of the layout segment. For this case, the storage device MUST ensure that all copies of the mirror are updated when any one of the mirrors is updated. If the storage device gets an error when updating one of the mirrors, then it MUST inform the client that the original WRITE had an error. The client then MUST inform the metadata server (see Section 8.2.3). The client's responsibility with respect to COMMIT is explained in Section 8.2.4. The client may choose any one of the mirrors and may use ffds_efficiency as described in Section 8.1 when making this choice.8.2.2. Client Updates All Mirrors
If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set, the client is responsible for updating all mirrored copies of the layout segments that it is given in the layout. A single failed update is sufficient to fail the entire operation. If all but one copy is updated successfully and the last one provides an error, then the client needs to inform the metadata server about the error. The client can use either LAYOUTRETURN or LAYOUTERROR to inform the metadata server that the update failed to that storage device. If the client is updating the mirrors serially, then it SHOULD stop at the first error encountered and report that to the metadata server. If the client is updating the mirrors in parallel, then it SHOULD wait until all storage devices respond so that it can report all errors encountered during the update.
8.2.3. Handling Write Errors
When the client reports a write error to the metadata server, the metadata server is responsible for determining if it wants to remove the errant mirror from the layout, if the mirror has recovered from some transient error, etc. When the client tries to get a new layout, the metadata server informs it of the decision by the contents of the layout. The client MUST NOT assume that the contents of the previous layout will match those of the new one. If it has updates that were not committed to all mirrors, then it MUST resend those updates to all mirrors. There is no provision in the protocol for the metadata server to directly determine that the client has or has not recovered from an error. For example, if a storage device was network partitioned from the client and the client reported the error to the metadata server, then the network partition would be repaired, and all of the copies would be successfully updated. There is no mechanism for the client to report that fact, and the metadata server is forced to repair the file across the mirror. If the client supports NFSv4.2, it can use LAYOUTERROR and LAYOUTRETURN to provide hints to the metadata server about the recovery efforts. A LAYOUTERROR on a file is for a non-fatal error. A subsequent LAYOUTRETURN without a ff_ioerr4 indicates that the client successfully replayed the I/O to all mirrors. Any LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server needs to repair. The client MUST be prepared for the LAYOUTERROR to trigger a CB_LAYOUTRECALL if the metadata server determines it needs to start repairing the file.8.2.4. Handling Write COMMITs
When stable writes are done to the metadata server or to a single replica (if allowed by the use of FF_FLAGS_WRITE_ONE_MIRROR), it is the responsibility of the receiving node to propagate the written data stably, before replying to the client. In the corresponding cases in which unstable writes are done, the receiving node does not have any such obligation, although it may choose to asynchronously propagate the updates. However, once a COMMIT is replied to, all replicas must reflect the writes that have been done, and this data must have been committed to stable storage on all replicas.
In order to avoid situations in which stale data is read from
replicas to which writes have not been propagated:
o A client that has outstanding unstable writes made to single node
(metadata server or storage device) MUST do all reads from that
same node.
o When writes are flushed to the server (for example, to implement
close-to-open semantics), a COMMIT must be done by the client to
ensure that up-to-date written data will be available irrespective
of the particular replica read.
8.3. Metadata Server Resilvering of the File
The metadata server may elect to create a new mirror of the layout
segments at any time. This might be to resilver a copy on a storage
device that was down for servicing, to provide a copy of the layout
segments on storage with different storage performance
characteristics, etc. As the client will not be aware of the new
mirror and the metadata server will not be aware of updates that the
client is making to the layout segments, the metadata server MUST
recall the writable layout segment(s) that it is resilvering. If the
client issues a LAYOUTGET for a writable layout segment that is in
the process of being resilvered, then the metadata server can deny
that request with an NFS4ERR_LAYOUTUNAVAILABLE. The client would
then have to perform the I/O through the metadata server.
9. Flexible File Layout Type Return
layoutreturn_file4 is used in the LAYOUTRETURN operation to convey
layout-type-specific information to the server. It is defined in
Section 18.44.1 of [RFC5661] as follows:
<CODE BEGINS>
/* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
const LAYOUT4_RET_REC_FILE = 1;
const LAYOUT4_RET_REC_FSID = 2;
const LAYOUT4_RET_REC_ALL = 3;
enum layoutreturn_type4 {
LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL
};
struct layoutreturn_file4 {
offset4 lrf_offset;
length4 lrf_length;
stateid4 lrf_stateid;
/* layouttype4 specific data */
opaque lrf_body<>;
};
union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
case LAYOUTRETURN4_FILE:
layoutreturn_file4 lr_layout;
default:
void;
};
struct LAYOUTRETURN4args {
/* CURRENT_FH: file */
bool lora_reclaim;
layouttype4 lora_layout_type;
layoutiomode4 lora_iomode;
layoutreturn4 lora_layoutreturn;
};
<CODE ENDS>
If the lora_layout_type layout type is LAYOUT4_FLEX_FILES and the
lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value
is defined by ff_layoutreturn4 (see Section 9.3). This allows the
client to report I/O error information or layout usage statistics
back to the metadata server as defined below. Note that while the
data structures are built on concepts introduced in NFSv4.2, the
effective discriminated union (lora_layout_type combined with
ff_layoutreturn4) allows for an NFSv4.1 metadata server to utilize
the data.
9.1. I/O Error Reporting
9.1.1. ff_ioerr4
<CODE BEGINS>
/// struct ff_ioerr4 {
/// offset4 ffie_offset;
/// length4 ffie_length;
/// stateid4 ffie_stateid;
/// device_error4 ffie_errors<>;
/// };
///
<CODE ENDS>
Recall that [RFC7862] defines device_error4 as: <CODE BEGINS> struct device_error4 { deviceid4 de_deviceid; nfsstat4 de_status; nfs_opnum4 de_opnum; }; <CODE ENDS> The ff_ioerr4 structure is used to return error indications for data files that generated errors during data transfers. These are hints to the metadata server that there are problems with that file. For each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length represent the storage device and byte range within the file in which the error occurred; ffie_errors represents the operation and type of error. The use of device_error4 is described in Section 15.6 of [RFC7862]. Even though the storage device might be accessed via NFSv3 and reports back NFSv3 errors to the client, the client is responsible for mapping these to appropriate NFSv4 status codes as de_status. Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 operations.9.2. Layout Usage Statistics
9.2.1. ff_io_latency4
<CODE BEGINS> /// struct ff_io_latency4 { /// uint64_t ffil_ops_requested; /// uint64_t ffil_bytes_requested; /// uint64_t ffil_ops_completed; /// uint64_t ffil_bytes_completed; /// uint64_t ffil_bytes_not_delivered; /// nfstime4 ffil_total_busy_time; /// nfstime4 ffil_aggregate_completion_time; /// }; /// <CODE ENDS>
Both operation counts and bytes transferred are kept in the ff_io_latency4. As seen in ff_layoutupdate4 (see Section 9.2.2), READ and WRITE operations are aggregated separately. READ operations are used for the ff_io_latency4 ffl_read. Both WRITE and COMMIT operations are used for the ff_io_latency4 ffl_write. "Requested" counters track what the client is attempting to do, and "completed" counters track what was done. There is no requirement that the client only report completed results that have matching requested results from the reported period. ffil_bytes_not_delivered is used to track the aggregate number of bytes requested but not fulfilled due to error conditions. ffil_total_busy_time is the aggregate time spent with outstanding RPC calls. ffil_aggregate_completion_time is the sum of all round-trip times for completed RPC calls. In Section 3.3.1 of [RFC5661], the nfstime4 is defined as the number of seconds and nanoseconds since midnight or zero hour January 1, 1970 Coordinated Universal Time (UTC). The use of nfstime4 in ff_io_latency4 is to store time since the start of the first I/O from the client after receiving the layout. In other words, these are to be decoded as duration and not as a date and time. Note that LAYOUTSTATS are cumulative, i.e., not reset each time the operation is sent. If two LAYOUTSTATS operations for the same file and layout stateid originate from the same NFS client and are processed at the same time by the metadata server, then the one containing the larger values contains the most recent time series data.9.2.2. ff_layoutupdate4
<CODE BEGINS> /// struct ff_layoutupdate4 { /// netaddr4 ffl_addr; /// nfs_fh4 ffl_fhandle; /// ff_io_latency4 ffl_read; /// ff_io_latency4 ffl_write; /// nfstime4 ffl_duration; /// bool ffl_local; /// }; /// <CODE ENDS>
ffl_addr differentiates which network address the client is connected to on the storage device. In the case of multipathing, ffl_fhandle indicates which read-only copy was selected. ffl_read and ffl_write convey the latencies for both READ and WRITE operations, respectively. ffl_duration is used to indicate the time period over which the statistics were collected. If true, ffl_local indicates that the I/O was serviced by the client's cache. This flag allows the client to inform the metadata server about "hot" access to a file it would not normally be allowed to report on.9.2.3. ff_iostats4
<CODE BEGINS> /// struct ff_iostats4 { /// offset4 ffis_offset; /// length4 ffis_length; /// stateid4 ffis_stateid; /// io_info4 ffis_read; /// io_info4 ffis_write; /// deviceid4 ffis_deviceid; /// ff_layoutupdate4 ffis_layoutupdate; /// }; /// <CODE ENDS> [RFC7862] defines io_info4 as: <CODE BEGINS> struct io_info4 { uint64_t ii_count; uint64_t ii_bytes; }; <CODE ENDS> With pNFS, data transfers are performed directly between the pNFS client and the storage devices. Therefore, the metadata server has no direct knowledge of the I/O operations being done and thus cannot create on its own statistical information about client I/O to optimize the data storage location. ff_iostats4 MAY be used by the client to report I/O statistics back to the metadata server upon returning the layout.
Since it is not feasible for the client to report every I/O that used the layout, the client MAY identify "hot" byte ranges for which to report I/O statistics. The definition and/or configuration mechanism of what is considered "hot" and the size of the reported byte range are out of the scope of this document. For client implementation, providing reasonable default values and an optional run-time management interface to control these parameters is suggested. For example, a client can define the default byte-range resolution to be 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 I/O operations per second. For each byte range, ffis_offset and ffis_length represent the starting offset of the range and the range length in bytes. ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and ffis_write.ii_bytes represent the number of contiguous READ and WRITE I/Os and the respective aggregate number of bytes transferred within the reported byte range. The combination of ffis_deviceid and ffl_addr uniquely identifies both the storage path and the network route to it. Finally, ffl_fhandle allows the metadata server to differentiate between multiple read-only copies of the file on the same storage device.9.3. ff_layoutreturn4
<CODE BEGINS> /// struct ff_layoutreturn4 { /// ff_ioerr4 fflr_ioerr_report<>; /// ff_iostats4 fflr_iostats_report<>; /// }; /// <CODE ENDS> When data file I/O operations fail, fflr_ioerr_report<> is used to report these errors to the metadata server as an array of elements of type ff_ioerr4. Each element in the array represents an error that occurred on the data file identified by ffie_errors.de_deviceid. If no errors are to be reported, the size of the fflr_ioerr_report<> array is set to zero. The client MAY also use fflr_iostats_report<> to report a list of I/O statistics as an array of elements of type ff_iostats4. Each element in the array represents statistics for a particular byte range. Byte ranges are not guaranteed to be disjoint and MAY repeat or intersect.
10. Flexible File Layout Type LAYOUTERROR
If the client is using NFSv4.2 to communicate with the metadata server, then instead of waiting for a LAYOUTRETURN to send error information to the metadata server (see Section 9.1), it MAY use LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that information. For the flexible file layout type, this means that LAYOUTERROR4args is treated the same as ff_ioerr4.11. Flexible File Layout Type LAYOUTSTATS
If the client is using NFSv4.2 to communicate with the metadata server, then instead of waiting for a LAYOUTRETURN to send I/O statistics to the metadata server (see Section 9.2), it MAY use LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that information. For the flexible file layout type, this means that LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same contents as in ffis_layoutupdate.12. Flexible File Layout Type Creation Hint
The layouthint4 type is defined in the [RFC5661] as follows: <CODE BEGINS> struct layouthint4 { layouttype4 loh_type; opaque loh_body<>; }; <CODE ENDS> The layouthint4 structure is used by the client to pass a hint about the type of layout it would like created for a particular file. If the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body opaque value is defined by the ff_layouthint4 type.12.1. ff_layouthint4
<CODE BEGINS> /// union ff_mirrors_hint switch (bool ffmc_valid) { /// case TRUE: /// uint32_t ffmc_mirrors; /// case FALSE: /// void; /// }; ///
/// struct ff_layouthint4 {
/// ff_mirrors_hint fflh_mirrors_hint;
/// };
///
<CODE ENDS>
This type conveys hints for the desired data map. All parameters are
optional so the client can give values for only the parameter it
cares about.
13. Recalling a Layout
While Section 12.5.5 of [RFC5661] discusses reasons independent of
layout type for recalling a layout, the flexible file layout type
metadata server should recall outstanding layouts in the following
cases:
o When the file's security policy changes, i.e., ACLs or permission
mode bits are set.
o When the file's layout changes, rendering outstanding layouts
invalid.
o When existing layouts are inconsistent with the need to enforce
locking constraints.
o When existing layouts are inconsistent with the requirements
regarding resilvering as described in Section 8.3.
13.1. CB_RECALL_ANY
The metadata server can use the CB_RECALL_ANY callback operation to
notify the client to return some or all of its layouts. Section 22.3
of [RFC5661] defines the allowed types of the "NFSv4 Recallable
Object Types Registry".
<CODE BEGINS>
/// const RCA4_TYPE_MASK_FF_LAYOUT_MIN = 16;
/// const RCA4_TYPE_MASK_FF_LAYOUT_MAX = 17;
///
struct CB_RECALL_ANY4args {
uint32_t craa_layouts_to_keep;
bitmap4 craa_type_mask;
};
<CODE ENDS>
Typically, CB_RECALL_ANY will be used to recall client state when the
server needs to reclaim resources. The craa_type_mask bitmap
specifies the type of resources that are recalled, and the
craa_layouts_to_keep value specifies how many of the recalled
flexible file layouts the client is allowed to keep. The mask flags
for the flexible file layout type are defined as follows:
<CODE BEGINS>
/// enum ff_cb_recall_any_mask {
/// PNFS_FF_RCA4_TYPE_MASK_READ = 16,
/// PNFS_FF_RCA4_TYPE_MASK_RW = 17
/// };
///
<CODE ENDS>
The flags represent the iomode of the recalled layouts. In response,
the client SHOULD return layouts of the recalled iomode that it needs
the least, keeping at most craa_layouts_to_keep flexible file
layouts.
The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return
layouts of iomode LAYOUTIOMODE4_READ. Similarly, the
PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client
is notified to return layouts of either iomode.
14. Client Fencing
In cases where clients are uncommunicative and their lease has
expired or when clients fail to return recalled layouts within a
lease period, the server MAY revoke client layouts and reassign these
resources to other clients (see Section 12.5.5 of [RFC5661]). To
avoid data corruption, the metadata server MUST fence off the revoked
clients from the respective data files as described in Section 2.2.
15. Security Considerations
The combination of components in a pNFS system is required to
preserve the security properties of NFSv4.1+ with respect to an
entity accessing data via a client. The pNFS feature partitions the
NFSv4.1+ file system protocol into two parts: the control protocol
and the data protocol. As the control protocol in this document is
NFS, the security properties are equivalent to the version of NFS
being used. The flexible file layout further divides the data
protocol into metadata and data paths. The security properties of the metadata path are equivalent to those of NFSv4.1x (see Sections 1.7.1 and 2.2.1 of [RFC5661]). And the security properties of the data path are equivalent to those of the version of NFS used to access the storage device, with the provision that the metadata server is responsible for authenticating client access to the data file. The metadata server provides appropriate credentials to the client to access data files on the storage device. It is also responsible for revoking access for a client to the storage device. The metadata server enforces the file access control policy at LAYOUTGET time. The client should use RPC authorization credentials for getting the layout for the requested iomode ((LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW), and the server verifies the permissions and ACL for these credentials, possibly returning NFS4ERR_ACCESS if the client is not allowed the requested iomode. If the LAYOUTGET operation succeeds, the client receives, as part of the layout, a set of credentials allowing it I/O access to the specified data files corresponding to the requested iomode. When the client acts on I/O operations on behalf of its local users, it MUST authenticate and authorize the user by issuing respective OPEN and ACCESS calls to the metadata server, similar to having NFSv4 data delegations. The combination of filehandle, synthetic uid, and gid in the layout is the way that the metadata server enforces access control to the data server. The client only has access to filehandles of file objects and not directory objects. Thus, given a filehandle in a layout, it is not possible to guess the parent directory filehandle. Further, as the data file permissions only allow the given synthetic uid read/write permission and the given synthetic gid read permission, knowing the synthetic ids of one file does not necessarily allow access to any other data file on the storage device. The metadata server can also deny access at any time by fencing the data file, which means changing the synthetic ids. In turn, that forces the client to return its current layout and get a new layout if it wants to continue I/O to the data file. If access is allowed, the client uses the corresponding (read-only or read/write) credentials to perform the I/O operations at the data file's storage devices. When the metadata server receives a request to change a file's permissions or ACL, it SHOULD recall all layouts for that file and then MUST fence off any clients still holding outstanding layouts for the respective files by implicitly invalidating the previously distributed credential on all data file comprising the file in question. It is REQUIRED that this be done before committing to the new permissions and/or ACL. By requesting
new layouts, the clients will reauthorize access against the modified access control metadata. Recalling the layouts in this case is intended to prevent clients from getting an error on I/Os done after the client was fenced off.15.1. RPCSEC_GSS and Security Services
Because of the special use of principals within the loosely coupled model, the issues are different depending on the coupling model.15.1.1. Loosely Coupled
RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] contains facilities that would allow it to be used to authorize the client to the storage device on behalf of the metadata server. Doing so would require that each of the metadata server, storage device, and client would need to implement RPCSEC_GSSv3 using an RPC-application-defined structured privilege assertion in a manner described in Section 4.9.1 of [RFC7862]. The specifics necessary to do so are not described in this document. This is principally because any such specification would require extensive implementation work on a wide range of storage devices, which would be unlikely to result in a widely usable specification for a considerable time. As a result, the layout type described in this document will not provide support for use of RPCSEC_GSS together with the loosely coupled model. However, future layout types could be specified, which would allow such support, either through the use of RPCSEC_GSSv3 or in other ways.15.1.2. Tightly Coupled
With tight coupling, the principal used to access the metadata file is exactly the same as used to access the data file. The storage device can use the control protocol to validate any RPC credentials. As a result, there are no security issues related to using RPCSEC_GSS with a tightly coupled system. For example, if Kerberos V5 Generic Security Service Application Program Interface (GSS-API) [RFC4121] is used as the security mechanism, then the storage device could use a control protocol to validate the RPC credentials to the metadata server.16. IANA Considerations
[RFC5661] introduced the "pNFS Layout Types Registry"; new layout type numbers in this registry need to be assigned by IANA. This document defines the protocol associated with an existing layout type number: LAYOUT4_FLEX_FILES. See Table 1.
+--------------------+------------+----------+-----+----------------+ | Layout Type Name | Value | RFC | How | Minor Versions | +--------------------+------------+----------+-----+----------------+ | LAYOUT4_FLEX_FILES | 0x00000004 | RFC 8435 | L | 1 | +--------------------+------------+----------+-----+----------------+ Table 1: Layout Type Assignments [RFC5661] also introduced the "NFSv4 Recallable Object Types Registry". This document defines new recallable objects for RCA4_TYPE_MASK_FF_LAYOUT_MIN and RCA4_TYPE_MASK_FF_LAYOUT_MAX (see Table 2). +------------------------------+-------+--------+-----+-------------+ | Recallable Object Type Name | Value | RFC | How | Minor | | | | | | Versions | +------------------------------+-------+--------+-----+-------------+ | RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFC | L | 1 | | | | 8435 | | | | RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFC | L | 1 | | | | 8435 | | | +------------------------------+-------+--------+-----+-------------+ Table 2: Recallable Object Type Assignments17. References
17.1. Normative References
[LEGAL] IETF Trust, "Trust Legal Provisions (TLP)", <https://trustee.ietf.org/trust-legal-provisions.html>. [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, DOI 10.17487/RFC1813, June 1995, <https://www.rfc-editor.org/info/rfc1813>. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. [RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism: Version 2", RFC 4121, DOI 10.17487/RFC4121, July 2005, <https://www.rfc-editor.org/info/rfc4121>.
[RFC4506] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 2006, <https://www.rfc-editor.org/info/rfc4506>. [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, May 2009, <https://www.rfc-editor.org/info/rfc5531>. [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, <https://www.rfc-editor.org/info/rfc5661>. [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, DOI 10.17487/RFC5662, January 2010, <https://www.rfc-editor.org/info/rfc5662>. [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, March 2015, <https://www.rfc-editor.org/info/rfc7530>. [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) Security Version 3", RFC 7861, DOI 10.17487/RFC7861, November 2016, <https://www.rfc-editor.org/info/rfc7861>. [RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862, November 2016, <https://www.rfc-editor.org/info/rfc7862>. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>. [RFC8434] Haynes, T., "Requirements for Parallel NFS (pNFS) Layout Types", RFC 8434, DOI 10.17487/RFC8434, August 2018, <https://www.rfc-editor.org/info/rfc8434>.17.2. Informative References
[RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol (LDAP): Schema for User Applications", RFC 4519, DOI 10.17487/RFC4519, June 2006, <https://www.rfc-editor.org/info/rfc4519>.
Acknowledgments
The following individuals provided miscellaneous comments to early draft versions of this document: Matt W. Benjamin, Adam Emerson, J. Bruce Fields, and Lev Solomonov. The following individuals provided miscellaneous comments to the final draft versions of this document: Anand Ganesh, Robert Wipfel, Gobikrishnan Sundharraj, Trond Myklebust, Rick Macklem, and Jim Sermersheim. Idan Kedar caught a nasty bug in the interaction of client-side mirroring and the minor versioning of devices. Dave Noveck provided comprehensive reviews of the document during the working group last calls. He also rewrote Section 2.3. Olga Kornievskaia made a convincing case against the use of a credential versus a principal in the fencing approach. Andy Adamson and Benjamin Kaduk helped to sharpen the focus. Benjamin Kaduk and Olga Kornievskaia also helped provide concrete scenarios for loosely coupled security mechanisms. In the end, Olga proved that as defined, the loosely coupled model would not work with RPCSEC_GSS. Tigran Mkrtchyan provided the use case for not allowing the client to proxy the I/O through the data server. Rick Macklem provided the use case for only writing to a single mirror.Authors' Addresses
Benny Halevy Email: bhalevy@gmail.com Thomas Haynes Hammerspace 4300 El Camino Real Ste 105 Los Altos, CA 94022 United States of America Email: loghyr@gmail.com