Tech-invite3GPPspaceIETFspace
959493929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 5661

Network File System (NFS) Version 4 Minor Version 1 Protocol

Pages: 617
Obsoleted by:  8881
Updated by:  81788434
Part 11 of 20 – Pages 309 to 334
First   Prev   Next

Top   ToC   RFC5661 - Page 309   prevText

13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type

This section describes the semantics and format of NFSv4.1 file-based layouts for pNFS. NFSv4.1 file-based layouts use the LAYOUT4_NFSV4_1_FILES layout type. The LAYOUT4_NFSV4_1_FILES type defines striping data across multiple NFSv4.1 data servers.

13.1. Client ID and Session Considerations

Sessions are a REQUIRED feature of NFSv4.1, and this extends to both the metadata server and file-based (NFSv4.1-based) data servers. The role a server plays in pNFS is determined by the result it returns from EXCHANGE_ID. The roles are: o Metadata server (EXCHGID4_FLAG_USE_PNFS_MDS is set in the result eir_flags). o Data server (EXCHGID4_FLAG_USE_PNFS_DS). o Non-metadata server (EXCHGID4_FLAG_USE_NON_PNFS). This is an NFSv4.1 server that does not support operations (e.g., LAYOUTGET) or attributes that pertain to pNFS. The client MAY request zero or more of EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_DS, or EXCHGID4_FLAG_USE_PNFS_MDS, even though some combinations (e.g., EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS) are contradictory. However, the server MUST only return the following acceptable combinations: +--------------------------------------------------------+ | Acceptable Results from EXCHANGE_ID | +--------------------------------------------------------+ | EXCHGID4_FLAG_USE_PNFS_MDS | | EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS | | EXCHGID4_FLAG_USE_PNFS_DS | | EXCHGID4_FLAG_USE_NON_PNFS | | EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS | +--------------------------------------------------------+ As the above table implies, a server can have one or two roles. A server can be both a metadata server and a data server, or it can be both a data server and non-metadata server. In addition to returning two roles in the EXCHANGE_ID's results, and thus serving both roles
Top   ToC   RFC5661 - Page 310
   via a common client ID, a server can serve two roles by returning a
   unique client ID and server owner for each role in each of two
   EXCHANGE_ID results, with each result indicating each role.

   In the case of a server with concurrent pNFS roles that are served by
   a common client ID, if the EXCHANGE_ID request from the client has
   zero or a combination of the bits set in eia_flags, the server result
   should set bits that represent the higher of the acceptable
   combination of the server roles, with a preference to match the roles
   requested by the client.  Thus, if a client request has
   (EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS |
   EXCHGID4_FLAG_USE_PNFS_DS) flags set, and the server is both a
   metadata server and a data server, serving both the roles by a common
   client ID, the server SHOULD return with (EXCHGID4_FLAG_USE_PNFS_MDS
   | EXCHGID4_FLAG_USE_PNFS_DS) set.

   In the case of a server that has multiple concurrent pNFS roles, each
   role served by a unique client ID, if the client specifies zero or a
   combination of roles in the request, the server results SHOULD return
   only one of the roles from the combination specified by the client
   request.  If the role specified by the server result does not match
   the intended use by the client, the client should send the
   EXCHANGE_ID specifying just the interested pNFS role.

   If a pNFS metadata client gets a layout that refers it to an NFSv4.1
   data server, it needs a client ID on that data server.  If it does
   not yet have a client ID from the server that had the
   EXCHGID4_FLAG_USE_PNFS_DS flag set in the EXCHANGE_ID results, then
   the client needs to send an EXCHANGE_ID to the data server, using the
   same co_ownerid as it sent to the metadata server, with the
   EXCHGID4_FLAG_USE_PNFS_DS flag set in the arguments.  If the server's
   EXCHANGE_ID results have EXCHGID4_FLAG_USE_PNFS_DS set, then the
   client may use the client ID to create sessions that will exchange
   pNFS data operations.  The client ID returned by the data server has
   no relationship with the client ID returned by a metadata server
   unless the client IDs are equal, and the server owners and server
   scopes of the data server and metadata server are equal.

   In NFSv4.1, the session ID in the SEQUENCE operation implies the
   client ID, which in turn might be used by the server to map the
   stateid to the right client/server pair.  However, when a data server
   is presented with a READ or WRITE operation with a stateid, because
   the stateid is associated with a client ID on a metadata server, and
   because the session ID in the preceding SEQUENCE operation is tied to
   the client ID of the data server, the data server has no obvious way
   to determine the metadata server from the COMPOUND procedure, and
Top   ToC   RFC5661 - Page 311
   thus has no way to validate the stateid.  One RECOMMENDED approach is
   for pNFS servers to encode metadata server routing and/or identity
   information in the data server filehandles as returned in the layout.

   If metadata server routing and/or identity information is encoded in
   data server filehandles, when the metadata server identity or
   location changes, the data server filehandles it gave out will become
   invalid (stale), and so the metadata server MUST first recall the
   layouts.  Invalidating a data server filehandle does not render the
   NFS client's data cache invalid.  The client's cache should map a
   data server filehandle to a metadata server filehandle, and a
   metadata server filehandle to cached data.

   If a server is both a metadata server and a data server, the server
   might need to distinguish operations on files that are directed to
   the metadata server from those that are directed to the data server.
   It is RECOMMENDED that the values of the filehandles returned by the
   LAYOUTGET operation be different than the value of the filehandle
   returned by the OPEN of the same file.

   Another scenario is for the metadata server and the storage device to
   be distinct from one client's point of view, and the roles reversed
   from another client's point of view.  For example, in the cluster
   file system model, a metadata server to one client might be a data
   server to another client.  If NFSv4.1 is being used as the storage
   protocol, then pNFS servers need to encode the values of filehandles
   according to their specific roles.

13.1.1. Sessions Considerations for Data Servers

Section 2.10.11.2 states that a client has to keep its lease renewed in order to prevent a session from being deleted by the server. If the reply to EXCHANGE_ID has just the EXCHGID4_FLAG_USE_PNFS_DS role set, then (as noted in Section 13.6) the client will not be able to determine the data server's lease_time attribute because GETATTR will not be permitted. Instead, the rule is that any time a client receives a layout referring it to a data server that returns just the EXCHGID4_FLAG_USE_PNFS_DS role, the client MAY assume that the lease_time attribute from the metadata server that returned the layout applies to the data server. Thus, the data server MUST be aware of the values of all lease_time attributes of all metadata servers for which it is providing I/O, and it MUST use the maximum of all such lease_time values as the lease interval for all client IDs and sessions established on it. For example, if one metadata server has a lease_time attribute of 20 seconds, and a second metadata server has a lease_time attribute of 10 seconds, then if both servers return layouts that refer to an
Top   ToC   RFC5661 - Page 312
   EXCHGID4_FLAG_USE_PNFS_DS-only data server, the data server MUST
   renew a client's lease if the interval between two SEQUENCE
   operations on different COMPOUND requests is less than 20 seconds.

13.2. File Layout Definitions

The following definitions apply to the LAYOUT4_NFSV4_1_FILES layout type and may be applicable to other layout types. Unit. A unit is a fixed-size quantity of data written to a data server. Pattern. A pattern is a method of distributing one or more equal sized units across a set of data servers. A pattern is iterated one or more times. Stripe. A stripe is a set of data distributed across a set of data servers in a pattern before that pattern repeats. Stripe Count. A stripe count is the number of units in a pattern. Stripe Width. A stripe width is the size of a stripe in bytes. The stripe width = the stripe count * the size of the stripe unit. Hereafter, this document will refer to a unit that is a written in a pattern as a "stripe unit". A pattern may have more stripe units than data servers. If so, some data servers will have more than one stripe unit per stripe. A data server that has multiple stripe units per stripe MAY store each unit in a different data file (and depending on the implementation, will possibly assign a unique data filehandle to each data file).

13.3. File Layout Data Types

The high level NFSv4.1 layout types are nfsv4_1_file_layouthint4, nfsv4_1_file_layout_ds_addr4, and nfsv4_1_file_layout4. The SETATTR operation supports a layout hint attribute (Section 5.12.4). When the client sets a layout hint (data type layouthint4) with a layout type of LAYOUT4_NFSV4_1_FILES (the loh_type field), the loh_body field contains a value of data type nfsv4_1_file_layouthint4.
Top   ToC   RFC5661 - Page 313
   const NFL4_UFLG_MASK            = 0x0000003F;
   const NFL4_UFLG_DENSE           = 0x00000001;
   const NFL4_UFLG_COMMIT_THRU_MDS = 0x00000002;
   const NFL4_UFLG_STRIPE_UNIT_SIZE_MASK
                                   = 0xFFFFFFC0;

   typedef uint32_t nfl_util4;

   enum filelayout_hint_care4 {
           NFLH4_CARE_DENSE        = NFL4_UFLG_DENSE,

           NFLH4_CARE_COMMIT_THRU_MDS
                                   = NFL4_UFLG_COMMIT_THRU_MDS,

           NFLH4_CARE_STRIPE_UNIT_SIZE
                                   = 0x00000040,

           NFLH4_CARE_STRIPE_COUNT = 0x00000080
   };

   /* Encoded in the loh_body field of data type layouthint4: */

   struct nfsv4_1_file_layouthint4 {
           uint32_t        nflh_care;
           nfl_util4       nflh_util;
           count4          nflh_stripe_count;
   };

   The generic layout hint structure is described in Section 3.3.19.
   The client uses the layout hint in the layout_hint (Section 5.12.4)
   attribute to indicate the preferred type of layout to be used for a
   newly created file.  The LAYOUT4_NFSV4_1_FILES layout-type-specific
   content for the layout hint is composed of three fields.  The first
   field, nflh_care, is a set of flags indicating which values of the
   hint the client cares about.  If the NFLH4_CARE_DENSE flag is set,
   then the client indicates in the second field, nflh_util, a
   preference for how the data file is packed (Section 13.4.4), which is
   controlled by the value of the expression nflh_util & NFL4_UFLG_DENSE
   ("&" represents the bitwise AND operator).  If the
   NFLH4_CARE_COMMIT_THRU_MDS flag is set, then the client indicates a
   preference for whether the client should send COMMIT operations to
   the metadata server or data server (Section 13.7), which is
   controlled by the value of nflh_util & NFL4_UFLG_COMMIT_THRU_MDS.  If
   the NFLH4_CARE_STRIPE_UNIT_SIZE flag is set, the client indicates its
   preferred stripe unit size, which is indicated in nflh_util &
   NFL4_UFLG_STRIPE_UNIT_SIZE_MASK (thus, the stripe unit size MUST be a
   multiple of 64 bytes).  The minimum stripe unit size is 64 bytes.  If
Top   ToC   RFC5661 - Page 314
   the NFLH4_CARE_STRIPE_COUNT flag is set, the client indicates in the
   third field, nflh_stripe_count, the stripe count.  The stripe count
   multiplied by the stripe unit size is the stripe width.

   When LAYOUTGET returns a LAYOUT4_NFSV4_1_FILES layout (indicated in
   the loc_type field of the lo_content field), the loc_body field of
   the lo_content field contains a value of data type
   nfsv4_1_file_layout4.  Among other content, nfsv4_1_file_layout4 has
   a storage device ID (field nfl_deviceid) of data type deviceid4.  The
   GETDEVICEINFO operation maps a device ID to a storage device address
   (type device_addr4).  When GETDEVICEINFO returns a device address
   with a layout type of LAYOUT4_NFSV4_1_FILES (the da_layout_type
   field), the da_addr_body field contains a value of data type
   nfsv4_1_file_layout_ds_addr4.

   typedef netaddr4 multipath_list4<>;

   /*
    * Encoded in the da_addr_body field of
    * data type device_addr4:
    */
   struct nfsv4_1_file_layout_ds_addr4 {
           uint32_t        nflda_stripe_indices<>;
           multipath_list4 nflda_multipath_ds_list<>;
   };

   The nfsv4_1_file_layout_ds_addr4 data type represents the device
   address.  It is composed of two fields:

   1.  nflda_multipath_ds_list: An array of lists of data servers, where
       each list can be one or more elements, and each element
       represents a data server address that may serve equally as the
       target of I/O operations (see Section 13.5).  The length of this
       array might be different than the stripe count.

   2.  nflda_stripe_indices: An array of indices used to index into
       nflda_multipath_ds_list.  The value of each element of
       nflda_stripe_indices MUST be less than the number of elements in
       nflda_multipath_ds_list.  Each element of nflda_multipath_ds_list
       SHOULD be referred to by one or more elements of
       nflda_stripe_indices.  The number of elements in
       nflda_stripe_indices is always equal to the stripe count.
Top   ToC   RFC5661 - Page 315
   /*
    * Encoded in the loc_body field of
    * data type layout_content4:
    */
   struct nfsv4_1_file_layout4 {
            deviceid4      nfl_deviceid;
            nfl_util4      nfl_util;
            uint32_t       nfl_first_stripe_index;
            offset4        nfl_pattern_offset;
            nfs_fh4        nfl_fh_list<>;
   };

   The nfsv4_1_file_layout4 data type represents the layout.  It is
   composed of the following fields:

   1.  nfl_deviceid: The device ID that maps to a value of type
       nfsv4_1_file_layout_ds_addr4.

   2.  nfl_util: Like the nflh_util field of data type
       nfsv4_1_file_layouthint4, a compact representation of how the
       data on a file on each data server is packed, whether the client
       should send COMMIT operations to the metadata server or data
       server, and the stripe unit size.  If a server returns two or
       more overlapping layouts, each stripe unit size in each
       overlapping layout MUST be the same.

   3.  nfl_first_stripe_index: The index into the first element of the
       nflda_stripe_indices array to use.

   4.  nfl_pattern_offset: This field is the logical offset into the
       file where the striping pattern starts.  It is required for
       converting the client's logical I/O offset (e.g., the current
       offset in a POSIX file descriptor before the read() or write()
       system call is sent) into the stripe unit number (see
       Section 13.4.1).

       If dense packing is used, then nfl_pattern_offset is also needed
       to convert the client's logical I/O offset to an offset on the
       file on the data server corresponding to the stripe unit number
       (see Section 13.4.4).

       Note that nfl_pattern_offset is not always the same as lo_offset.
       For example, via the LAYOUTGET operation, a client might request
       a layout starting at offset 1000 of a file that has its striping
       pattern start at offset zero.
Top   ToC   RFC5661 - Page 316
   5.  nfl_fh_list: An array of data server filehandles for each list of
       data servers in each element of the nflda_multipath_ds_list
       array.  The number of elements in nfl_fh_list depends on whether
       sparse or dense packing is being used.

       *  If sparse packing is being used, the number of elements in
          nfl_fh_list MUST be one of three values:

          +  Zero.  This means that filehandles used for each data
             server are the same as the filehandle returned by the OPEN
             operation from the metadata server.

          +  One.  This means that every data server uses the same
             filehandle: what is specified in nfl_fh_list[0].

          +  The same number of elements in nflda_multipath_ds_list.
             Thus, in this case, when sending an I/O operation to any
             data server in nflda_multipath_ds_list[X], the filehandle
             in nfl_fh_list[X] MUST be used.

          See the discussion on sparse packing in Section 13.4.4.


       *  If dense packing is being used, the number of elements in
          nfl_fh_list MUST be the same as the number of elements in
          nflda_stripe_indices.  Thus, when sending an I/O operation to
          any data server in
          nflda_multipath_ds_list[nflda_stripe_indices[Y]], the
          filehandle in nfl_fh_list[Y] MUST be used.  In addition, any
          time there exists i and j, (i != j), such that the
          intersection of
          nflda_multipath_ds_list[nflda_stripe_indices[i]] and
          nflda_multipath_ds_list[nflda_stripe_indices[j]] is not empty,
          then nfl_fh_list[i] MUST NOT equal nfl_fh_list[j].  In other
          words, when dense packing is being used, if a data server
          appears in two or more units of a striping pattern, each
          reference to the data server MUST use a different filehandle.

          Indeed, if there are multiple striping patterns, as indicated
          by the presence of multiple objects of data type layout4
          (either returned in one or multiple LAYOUTGET operations), and
          a data server is the target of a unit of one pattern and
          another unit of another pattern, then each reference to each
          data server MUST use a different filehandle.

          See the discussion on dense packing in Section 13.4.4.

   The details on the interpretation of the layout are in Section 13.4.
Top   ToC   RFC5661 - Page 317

13.4. Interpreting the File Layout

13.4.1. Determining the Stripe Unit Number

To find the stripe unit number that corresponds to the client's logical file offset, the pattern offset will also be used. The i'th stripe unit (SUi) is: relative_offset = file_offset - nfl_pattern_offset; SUi = floor(relative_offset / stripe_unit_size);

13.4.2. Interpreting the File Layout Using Sparse Packing

When sparse packing is used, the algorithm for determining the filehandle and set of data-server network addresses to write stripe unit i (SUi) to is: stripe_count = number of elements in nflda_stripe_indices; j = (SUi + nfl_first_stripe_index) % stripe_count; idx = nflda_stripe_indices[j]; fh_count = number of elements in nfl_fh_list; ds_count = number of elements in nflda_multipath_ds_list; switch (fh_count) { case ds_count: fh = nfl_fh_list[idx]; break; case 1: fh = nfl_fh_list[0]; break; case 0: fh = filehandle returned by OPEN; break; default: throw a fatal exception; break; } address_list = nflda_multipath_ds_list[idx];
Top   ToC   RFC5661 - Page 318
   The client would then select a data server from address_list, and
   send a READ or WRITE operation using the filehandle specified in fh.

   Consider the following example:

   Suppose we have a device address consisting of seven data servers,
   arranged in three equivalence (Section 13.5) classes:

      { A, B, C, D }, { E }, { F, G }

   where A through G are network addresses.

   Then

      nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

   i.e.,

      nflda_multipath_ds_list[0] = { A, B, C, D }

      nflda_multipath_ds_list[1] = { E }

      nflda_multipath_ds_list[2] = { F, G }

   Suppose the striping index array is:

      nflda_stripe_indices<> = { 2, 0, 1, 0 }

   Now suppose the client gets a layout that has a device ID that maps
   to the above device address.  The initial index contains

      nfl_first_stripe_index = 2,

   and the filehandle list is

      nfl_fh_list = { 0x36, 0x87, 0x67 }.

   If the client wants to write to SU0, the set of valid { network
   address, filehandle } combinations for SUi are determined by:

      nfl_first_stripe_index = 2
Top   ToC   RFC5661 - Page 319
   So

      idx = nflda_stripe_indices[(0 + 2) % 4]

         = nflda_stripe_indices[2]

         = 1

   So

      nflda_multipath_ds_list[1] = { E }

   and

      nfl_fh_list[1] = { 0x87 }

   The client can thus write SU0 to { 0x87, { E } }.

   The destinations of the first 13 storage units are:

                    +-----+------------+--------------+
                    | SUi | filehandle | data servers |
                    +-----+------------+--------------+
                    | 0   | 87         | E            |
                    | 1   | 36         | A,B,C,D      |
                    | 2   | 67         | F,G          |
                    | 3   | 36         | A,B,C,D      |
                    | 4   | 87         | E            |
                    | 5   | 36         | A,B,C,D      |
                    | 6   | 67         | F,G          |
                    | 7   | 36         | A,B,C,D      |
                    | 8   | 87         | E            |
                    | 9   | 36         | A,B,C,D      |
                    | 10  | 67         | F,G          |
                    | 11  | 36         | A,B,C,D      |
                    | 12  | 87         | E            |
                    +-----+------------+--------------+

13.4.3. Interpreting the File Layout Using Dense Packing

When dense packing is used, the algorithm for determining the filehandle and set of data server network addresses to write stripe unit i (SUi) to is:
Top   ToC   RFC5661 - Page 320
      stripe_count = number of elements in nflda_stripe_indices;

      j = (SUi + nfl_first_stripe_index) % stripe_count;

      idx = nflda_stripe_indices[j];

      fh_count = number of elements in nfl_fh_list;
      ds_count = number of elements in nflda_multipath_ds_list;

      switch (fh_count) {
        case stripe_count:
          fh = nfl_fh_list[j];
          break;

        default:
          throw a fatal exception;
          break;
      }

      address_list = nflda_multipath_ds_list[idx];


   The client would then select a data server from address_list, and
   send a READ or WRITE operation using the filehandle specified in fh.

   Consider the following example (which is the same as the sparse
   packing example, except for the filehandle list):

   Suppose we have a device address consisting of seven data servers,
   arranged in three equivalence (Section 13.5) classes:

      { A, B, C, D }, { E }, { F, G }

   where A through G are network addresses.

   Then

      nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

   i.e.,

      nflda_multipath_ds_list[0] = { A, B, C, D }

      nflda_multipath_ds_list[1] = { E }

      nflda_multipath_ds_list[2] = { F, G }
Top   ToC   RFC5661 - Page 321
   Suppose the striping index array is:

      nflda_stripe_indices<> = { 2, 0, 1, 0 }

   Now suppose the client gets a layout that has a device ID that maps
   to the above device address.  The initial index contains

      nfl_first_stripe_index = 2,

   and

      nfl_fh_list = { 0x67, 0x37, 0x87, 0x36 }.

   The interesting examples for dense packing are SU1 and SU3 because
   each stripe unit refers to the same data server list, yet each stripe
   unit MUST use a different filehandle.  If the client wants to write
   to SU1, the set of valid { network address, filehandle } combinations
   for SUi are determined by:

      nfl_first_stripe_index = 2

   So

      j = (1 + 2) % 4 = 3

         idx = nflda_stripe_indices[j]

         = nflda_stripe_indices[3]

         = 0

   So

      nflda_multipath_ds_list[0] = { A, B, C, D }

   and

      nfl_fh_list[3] = { 0x36 }

   The client can thus write SU1 to { 0x36, { A, B, C, D } }.

   For SU3, j = (3 + 2) % 4 = 1, and nflda_stripe_indices[1] = 0.  Then
   nflda_multipath_ds_list[0] = { A, B, C, D }, and nfl_fh_list[1] =
   0x37.  The client can thus write SU3 to { 0x37, { A, B, C, D } }.
Top   ToC   RFC5661 - Page 322
   The destinations of the first 13 storage units are:

                    +-----+------------+--------------+
                    | SUi | filehandle | data servers |
                    +-----+------------+--------------+
                    | 0   | 87         | E            |
                    | 1   | 36         | A,B,C,D      |
                    | 2   | 67         | F,G          |
                    | 3   | 37         | A,B,C,D      |
                    | 4   | 87         | E            |
                    | 5   | 36         | A,B,C,D      |
                    | 6   | 67         | F,G          |
                    | 7   | 37         | A,B,C,D      |
                    | 8   | 87         | E            |
                    | 9   | 36         | A,B,C,D      |
                    | 10  | 67         | F,G          |
                    | 11  | 37         | A,B,C,D      |
                    | 12  | 87         | E            |
                    +-----+------------+--------------+

13.4.4. Sparse and Dense Stripe Unit Packing

The flag NFL4_UFLG_DENSE of the nfl_util4 data type (field nflh_util of the data type nfsv4_1_file_layouthint4 and field nfl_util of data type nfsv4_1_file_layout_ds_addr4) specifies how the data is packed within the data file on a data server. It allows for two different data packings: sparse and dense. The packing type determines the calculation that will be made to map the client-visible file offset to the offset within the data file located on the data server. If nfl_util & NFL4_UFLG_DENSE is zero, this means that sparse packing is being used. Hence, the logical offsets of the file as viewed by a client sending READs and WRITEs directly to the metadata server are the same offsets each data server uses when storing a stripe unit. The effect then, for striping patterns consisting of at least two stripe units, is for each data server file to be sparse or "holey". So for example, suppose there is a pattern with three stripe units, the stripe unit size is 4096 bytes, and there are three data servers in the pattern. Then, the file in data server 1 will have stripe units 0, 3, 6, 9, ... filled; data server 2's file will have stripe units 1, 4, 7, 10, ... filled; and data server 3's file will have stripe units 2, 5, 8, 11, ... filled. The unfilled stripe units of each file will be holes; hence, the files in each data server are sparse. If sparse packing is being used and a client attempts I/O to one of the holes, then an error MUST be returned by the data server. Using the above example, if data server 3 received a READ or WRITE
Top   ToC   RFC5661 - Page 323
   operation for block 4, the data server would return
   NFS4ERR_PNFS_IO_HOLE.  Thus, data servers need to understand the
   striping pattern in order to support sparse packing.

   If nfl_util & NFL4_UFLG_DENSE is one, this means that dense packing
   is being used, and the data server files have no holes.  Dense
   packing might be selected because the data server does not
   (efficiently) support holey files or because the data server cannot
   recognize read-ahead unless there are no holes.  If dense packing is
   indicated in the layout, the data files will be packed.  Using the
   same striping pattern and stripe unit size that were used for the
   sparse packing example, the corresponding dense packing example would
   have all stripe units of all data files filled as follows:

   o  Logical stripe units 0, 3, 6, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 1.

   o  Logical stripe units 1, 4, 7, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 2.

   o  Logical stripe units 2, 5, 8, ... of the file would live on stripe
      units 0, 1, 2, ... of the file of data server 3.

   Because dense packing does not leave holes on the data servers, the
   pNFS client is allowed to write to any offset of any data file of any
   data server in the stripe.  Thus, the data servers need not know the
   file's striping pattern.

   The calculation to determine the byte offset within the data file for
   dense data server layouts is:

      stripe_width = stripe_unit_size * N;
         where N = number of elements in nflda_stripe_indices.

      relative_offset = file_offset - nfl_pattern_offset;

      data_file_offset = floor(relative_offset / stripe_width)
         * stripe_unit_size
         + relative_offset % stripe_unit_size

   If dense packing is being used, and a data server appears more than
   once in a striping pattern, then to distinguish one stripe unit from
   another, the data server MUST use a different filehandle.  Let's
   suppose there are two data servers.  Logical stripe units 0, 3, 6 are
   served by data server 1; logical stripe units 1, 4, 7 are served by
   data server 2; and logical stripe units 2, 5, 8 are also served by
   data server 2.  Unless data server 2 has two filehandles (each
   referring to a different data file), then, for example, a write to
Top   ToC   RFC5661 - Page 324
   logical stripe unit 1 overwrites the write to logical stripe unit 2
   because both logical stripe units are located in the same stripe unit
   (0) of data server 2.

13.5. Data Server Multipathing

The NFSv4.1 file layout supports multipathing to multiple data server addresses. Data-server-level multipathing is used for bandwidth scaling via trunking (Section 2.10.5) and for higher availability of use in the case of a data-server failure. Multipathing allows the client to switch to another data server address which may be that of another data server that is exporting the same data stripe unit, without having to contact the metadata server for a new layout. To support data server multipathing, each element of the nflda_multipath_ds_list contains an array of one more data server network addresses. This array (data type multipath_list4) represents a list of data servers (each identified by a network address), with the possibility that some data servers will appear in the list multiple times. The client is free to use any of the network addresses as a destination to send data server requests. If some network addresses are less optimal paths to the data than others, then the MDS SHOULD NOT include those network addresses in an element of nflda_multipath_ds_list. If less optimal network addresses exist to provide failover, the RECOMMENDED method to offer the addresses is to provide them in a replacement device-ID-to-device-address mapping, or a replacement device ID. When a client finds that no data server in an element of nflda_multipath_ds_list responds, it SHOULD send a GETDEVICEINFO to attempt to replace the existing device-ID-to-device- address mappings. If the MDS detects that all data servers represented by an element of nflda_multipath_ds_list are unavailable, the MDS SHOULD send a CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID notifications for changed device IDs) to change the device-ID-to-device-address mappings to the available data servers. If the device ID itself will be replaced, the MDS SHOULD recall all layouts with the device ID, and thus force the client to get new layouts and device ID mappings via LAYOUTGET and GETDEVICEINFO. Generally, if two network addresses appear in an element of nflda_multipath_ds_list, they will designate the same data server, and the two data server addresses will support the implementation of client ID or session trunking (the latter is RECOMMENDED) as defined in Section 2.10.5. The two data server addresses will share the same server owner or major ID of the server owner. It is not always
Top   ToC   RFC5661 - Page 325
   necessary for the two data server addresses to designate the same
   server with trunking being used.  For example, the data could be
   read-only, and the data consist of exact replicas.

13.6. Operations Sent to NFSv4.1 Data Servers

Clients accessing data on an NFSv4.1 data server MUST send only the NULL procedure and COMPOUND procedures whose operations are taken only from two restricted subsets of the operations defined as valid NFSv4.1 operations. Clients MUST use the filehandle specified by the layout when accessing data on NFSv4.1 data servers. The first of these operation subsets consists of management operations. This subset consists of the BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION, DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID, SECINFO_NO_NAME, SET_SSV, and SEQUENCE operations. The client may use these operations in order to set up and maintain the appropriate client IDs, sessions, and security contexts involved in communication with the data server. Henceforth, these will be referred to as data-server housekeeping operations. The second subset consists of COMMIT, READ, WRITE, and PUTFH. These operations MUST be used with a current filehandle specified by the layout. In the case of PUTFH, the new current filehandle MUST be one taken from the layout. Henceforth, these will be referred to as data-server I/O operations. As described in Section 12.5.1, a client MUST NOT send an I/O to a data server for which it does not hold a valid layout; the data server MUST reject such an I/O. Unless the server has a concurrent non-data-server personality -- i.e., EXCHANGE_ID results returned (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_PNFS_MDS) or (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS) see Section 13.1 -- any attempted use of operations against a data server other than those specified in the two subsets above MUST return NFS4ERR_NOTSUPP to the client. When the server has concurrent data-server and non-data-server personalities, each COMPOUND sent by the client MUST be constructed so that it is appropriate to one of the two personalities, and it MUST NOT contain operations directed to a mix of those personalities. The server MUST enforce this. To understand the constraints, operations within a COMPOUND are divided into the following three classes: 1. An operation that is ambiguous regarding its personality assignment. This includes all of the data-server housekeeping operations. Additionally, if the server has assigned filehandles so that the ones defined by the layout are the same as those used
Top   ToC   RFC5661 - Page 326
       by the metadata server, all operations using such filehandles are
       within this class, with the following exception.  The exception
       is that if the operation uses a stateid that is incompatible with
       a data-server personality (e.g., a special stateid or the stateid
       has a non-zero "seqid" field, see Section 13.9.1), the operation
       is in class 3, as described below.  A COMPOUND containing
       multiple class 1 operations (and operations of no other class)
       MAY be sent to a server with multiple concurrent data server and
       non-data-server personalities.

   2.  An operation that is unambiguously referable to the data-server
       personality.  This includes data-server I/O operations where the
       filehandle is one that can only be validly directed to the data-
       server personality.

   3.  An operation that is unambiguously referable to the non-data-
       server personality.  This includes all COMPOUND operations that
       are neither data-server housekeeping nor data-server I/O
       operations, plus data-server I/O operations where the current fh
       (or the one to be made the current fh in the case of PUTFH) is
       only valid on the metadata server or where a stateid is used that
       is incompatible with the data server, i.e., is a special stateid
       or has a non-zero seqid value.

   When a COMPOUND first executes an operation from class 3 above, it
   acts as a normal COMPOUND on any other server, and the data-server
   personality ceases to be relevant.  There are no special restrictions
   on the operations in the COMPOUND to limit them to those for a data
   server.  When a PUTFH is done, filehandles derived from the layout
   are not valid.  If their format is not normally acceptable, then
   NFS4ERR_BADHANDLE MUST result.  Similarly, current filehandles for
   other operations do not accept filehandles derived from layouts and
   are not normally usable on the metadata server.  Using these will
   result in NFS4ERR_STALE.

   When a COMPOUND first executes an operation from class 2, which would
   be PUTFH where the filehandle is one from a layout, the COMPOUND
   henceforth is interpreted with respect to the data-server
   personality.  Operations outside the two classes discussed above MUST
   result in NFS4ERR_NOTSUPP.  Filehandles are validated using the rules
   of the data server, resulting in NFS4ERR_BADHANDLE and/or
   NFS4ERR_STALE even when they would not normally do so when addressed
   to the non-data-server personality.  Stateids must obey the rules of
   the data server in that any use of special stateids or stateids with
   non-zero seqid values must result in NFS4ERR_BAD_STATEID.
Top   ToC   RFC5661 - Page 327
   Until the server first executes an operation from class 2 or class 3,
   the client MUST NOT depend on the operation being executed by either
   the data-server or the non-data-server personality.  The server MUST
   pick one personality consistently for a given COMPOUND, with the only
   possible transition being a single one when the first operation from
   class 2 or class 3 is executed.

   Because of the complexity induced by assigning filehandles so they
   can be used on both a data server and a metadata server, it is
   RECOMMENDED that where the same server can have both personalities,
   the server assign separate unique filehandles to both personalities.
   This makes it unambiguous for which server a given request is
   intended.

   GETATTR and SETATTR MUST be directed to the metadata server.  In the
   case of a SETATTR of the size attribute, the control protocol is
   responsible for propagating size updates/truncations to the data
   servers.  In the case of extending WRITEs to the data servers, the
   new size must be visible on the metadata server once a LAYOUTCOMMIT
   has completed (see Section 12.5.4.2).  Section 13.10 describes the
   mechanism by which the client is to handle data-server files that do
   not reflect the metadata server's size.

13.7. COMMIT through Metadata Server

The file layout provides two alternate means of providing for the commit of data written through data servers. The flag NFL4_UFLG_COMMIT_THRU_MDS in the field nfl_util of the file layout (data type nfsv4_1_file_layout4) is an indication from the metadata server to the client of the REQUIRED way of performing COMMIT, either by sending the COMMIT to the data server or the metadata server. These two methods of dealing with the issue correspond to broad styles of implementation for a pNFS server supporting the file layout type. o When the flag is FALSE, COMMIT operations MUST to be sent to the data server to which the corresponding WRITE operations were sent. This approach is sometimes useful when file striping is implemented within the pNFS server (instead of the file system), with the individual data servers each implementing their own file systems. o When the flag is TRUE, COMMIT operations MUST be sent to the metadata server, rather than to the individual data servers. This approach is sometimes useful when file striping is implemented within the clustered file system that is the backend to the pNFS server. In such an implementation, each COMMIT to each data server might result in repeated writes of metadata blocks to the
Top   ToC   RFC5661 - Page 328
      detriment of write performance.  Sending a single COMMIT to the
      metadata server can be more efficient when there exists a
      clustered file system capable of implementing such a coordinated
      COMMIT.

      If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE, then in order to
      maintain the current NFSv4.1 commit and recovery model, the data
      servers MUST return a common writeverf verifier in all WRITE
      responses for a given file layout, and the metadata server's
      COMMIT implementation must return the same writeverf.  The value
      of the writeverf verifier MUST be changed at the metadata server
      or any data server that is referenced in the layout, whenever
      there is a server event that can possibly lead to loss of
      uncommitted data.  The scope of the verifier can be for a file or
      for the entire pNFS server.  It might be more difficult for the
      server to maintain the verifier at the file level, but the benefit
      is that only events that impact a given file will require recovery
      action.

   Note that if the layout specified dense packing, then the offset used
   to a COMMIT to the MDS may differ than that of an offset used to a
   COMMIT to the data server.

   The single COMMIT to the metadata server will return a verifier, and
   the client should compare it to all the verifiers from the WRITEs and
   fail the COMMIT if there are any mismatched verifiers.  If COMMIT to
   the metadata server fails, the client should re-send WRITEs for all
   the modified data in the file.  The client should treat modified data
   with a mismatched verifier as a WRITE failure and try to recover by
   resending the WRITEs to the original data server or using another
   path to that data if the layout has not been recalled.
   Alternatively, the client can obtain a new layout or it could rewrite
   the data directly to the metadata server.  If nfl_util &
   NFL4_UFLG_COMMIT_THRU_MDS is FALSE, sending a COMMIT to the metadata
   server might have no effect.  If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS
   is FALSE, a COMMIT sent to the metadata server should be used only to
   commit data that was written to the metadata server.  See
   Section 12.7.6 for recovery options.

13.8. The Layout Iomode

The layout iomode need not be used by the metadata server when servicing NFSv4.1 file-based layouts, although in some circumstances it may be useful. For example, if the server implementation supports reading from read-only replicas or mirrors, it would be useful for the server to return a layout enabling the client to do so. As such, the client SHOULD set the iomode based on its intent to read or write the data. The client may default to an iomode of LAYOUTIOMODE4_RW.
Top   ToC   RFC5661 - Page 329
   The iomode need not be checked by the data servers when clients
   perform I/O.  However, the data servers SHOULD still validate that
   the client holds a valid layout and return an error if the client
   does not.

13.9. Metadata and Data Server State Coordination

13.9.1. Global Stateid Requirements

When the client sends I/O to a data server, the stateid used MUST NOT be a layout stateid as returned by LAYOUTGET or sent by CB_LAYOUTRECALL. Permitted stateids are based on one of the following: an OPEN stateid (the stateid field of data type OPEN4resok as returned by OPEN), a delegation stateid (the stateid field of data types open_read_delegation4 and open_write_delegation4 as returned by OPEN or WANT_DELEGATION, or as sent by CB_PUSH_DELEG), or a stateid returned by the LOCK or LOCKU operations. The stateid sent to the data server MUST be sent with the seqid set to zero, indicating the most current version of that stateid, rather than indicating a specific non-zero seqid value. In no case is the use of special stateid values allowed. The stateid used for I/O MUST have the same effect and be subject to the same validation on a data server as it would if the I/O was being performed on the metadata server itself in the absence of pNFS. This has the implication that stateids are globally valid on both the metadata and data servers. This requires the metadata server to propagate changes in LOCK and OPEN state to the data servers, so that the data servers can validate I/O accesses. This is discussed further in Section 13.9.2. Depending on when stateids are propagated, the existence of a valid stateid on the data server may act as proof of a valid layout. Clients performing I/O operations need to select an appropriate stateid based on the locks (including opens and delegations) held by the client and the various types of state-owners sending the I/O requests. The rules for doing so when referencing data servers are somewhat different from those discussed in Section 8.2.5, which apply when accessing metadata servers. The following rules, applied in order of decreasing priority, govern the selection of the appropriate stateid: o If the client holds a delegation for the file in question, the delegation stateid should be used.
Top   ToC   RFC5661 - Page 330
   o  Otherwise, there must be an OPEN stateid for the current open-
      owner, and that OPEN stateid for the open file in question is
      used, unless mandatory locking prevents that.  See below.

   o  If the data server had previously responded with NFS4ERR_LOCKED to
      use of the OPEN stateid, then the client should use the byte-range
      lock stateid whenever one exists for that open file with the
      current lock-owner.

   o  Special stateids should never be used.  If they are used, the data
      server MUST reject the I/O with an NFS4ERR_BAD_STATEID error.

13.9.2. Data Server State Propagation

Since the metadata server, which handles byte-range lock and open- mode state changes as well as ACLs, might not be co-located with the data servers where I/O accesses are validated, the server implementation MUST take care of propagating changes of this state to the data servers. Once the propagation to the data servers is complete, the full effect of those changes MUST be in effect at the data servers. However, some state changes need not be propagated immediately, although all changes SHOULD be propagated promptly. These state propagations have an impact on the design of the control protocol, even though the control protocol is outside of the scope of this specification. Immediate propagation refers to the synchronous propagation of state from the metadata server to the data server(s); the propagation must be complete before returning to the client.
13.9.2.1. Lock State Propagation
If the pNFS server supports mandatory byte-range locking, any mandatory byte-range locks on a file MUST be made effective at the data servers before the request that establishes them returns to the caller. The effect MUST be the same as if the mandatory byte-range lock state were synchronously propagated to the data servers, even though the details of the control protocol may avoid actual transfer of the state under certain circumstances. On the other hand, since advisory byte-range lock state is not used for checking I/O accesses at the data servers, there is no semantic reason for propagating advisory byte-range lock state to the data servers. Since updates to advisory locks neither confer nor remove privileges, these changes need not be propagated immediately, and may not need to be propagated promptly. The updates to advisory locks need only be propagated when the data server needs to resolve a question about a stateid. In fact, if byte-range locking is not mandatory (i.e., is advisory) the clients are advised to avoid using
Top   ToC   RFC5661 - Page 331
   the byte-range lock-based stateids for I/O.  The stateids returned by
   OPEN are sufficient and eliminate overhead for this kind of state
   propagation.

   If a client gets back an NFS4ERR_LOCKED error from a data server,
   this is an indication that mandatory byte-range locking is in force.
   The client recovers from this by getting a byte-range lock that
   covers the affected range and re-sends the I/O with the stateid of
   the byte-range lock.

13.9.2.2. Open and Deny Mode Validation
Open and deny mode validation MUST be performed against the open and deny mode(s) held by the data servers. When access is reduced or a deny mode made more restrictive (because of CLOSE or OPEN_DOWNGRADE), the data server MUST prevent any I/Os that would be denied if performed on the metadata server. When access is expanded, the data server MUST make sure that no requests are subsequently rejected because of open or deny issues that no longer apply, given the previous relaxation.
13.9.2.3. File Attributes
Since the SETATTR operation has the ability to modify state that is visible on both the metadata and data servers (e.g., the size), care must be taken to ensure that the resultant state across the set of data servers is consistent, especially when truncating or growing the file. As described earlier, the LAYOUTCOMMIT operation is used to ensure that the metadata is synchronized with changes made to the data servers. For the NFSv4.1-based data storage protocol, it is necessary to re-synchronize state such as the size attribute, and the setting of mtime/change/atime. See Section 12.5.4 for a full description of the semantics regarding LAYOUTCOMMIT and attribute synchronization. It should be noted that by using an NFSv4.1-based layout type, it is possible to synchronize this state before LAYOUTCOMMIT occurs. For example, the control protocol can be used to query the attributes present on the data servers. Any changes to file attributes that control authorization or access as reflected by ACCESS calls or READs and WRITEs on the metadata server, MUST be propagated to the data servers for enforcement on READ and WRITE I/O calls. If the changes made on the metadata server result in more restrictive access permissions for any user, those changes MUST be propagated to the data servers synchronously.
Top   ToC   RFC5661 - Page 332
   The OPEN operation (Section 18.16.4) does not impose any requirement
   that I/O operations on an open file have the same credentials as the
   OPEN itself (unless EXCHGID4_FLAG_BIND_PRINC_STATEID is set when
   EXCHANGE_ID creates the client ID), and so it requires the server's
   READ and WRITE operations to perform appropriate access checking.
   Changes to ACLs also require new access checking by READ and WRITE on
   the server.  The propagation of access-right changes due to changes
   in ACLs may be asynchronous only if the server implementation is able
   to determine that the updated ACL is not more restrictive for any
   user specified in the old ACL.  Due to the relative infrequency of
   ACL updates, it is suggested that all changes be propagated
   synchronously.

13.10. Data Server Component File Size

A potential problem exists when a component data file on a particular data server has grown past EOF; the problem exists for both dense and sparse layouts. Imagine the following scenario: a client creates a new file (size == 0) and writes to byte 131072; the client then seeks to the beginning of the file and reads byte 100. The client should receive zeroes back as a result of the READ. However, if the striping pattern directs the client to send the READ to a data server other than the one that received the client's original WRITE, the data server servicing the READ may believe that the file's size is still 0 bytes. In that event, the data server's READ response will contain zero bytes and an indication of EOF. The data server can only return zeroes if it knows that the file's size has been extended. This would require the immediate propagation of the file's size to all data servers, which is potentially very costly. Therefore, the client that has initiated the extension of the file's size MUST be prepared to deal with these EOF conditions. When the offset in the arguments to READ is less than the client's view of the file size, if the READ response indicates EOF and/or contains fewer bytes than requested, the client will interpret such a response as a hole in the file, and the NFS client will substitute zeroes for the data. The NFSv4.1 protocol only provides close-to-open file data cache semantics; meaning that when the file is closed, all modified data is written to the server. When a subsequent OPEN of the file is done, the change attribute is inspected for a difference from a cached value for the change attribute. For the case above, this means that a LAYOUTCOMMIT will be done at close (along with the data WRITEs) and will update the file's size and change attribute. Access from another client after that point will result in the appropriate size being returned.
Top   ToC   RFC5661 - Page 333

13.11. Layout Revocation and Fencing

As described in Section 12.7, the layout-type-specific storage protocol is responsible for handling the effects of I/Os that started before lease expiration and extend through lease expiration. The LAYOUT4_NFSV4_1_FILES layout type can prevent all I/Os to data servers from being executed after lease expiration (this prevention is called "fencing"), without relying on a precise client lease timer and without requiring data servers to maintain lease timers. The LAYOUT4_NFSV4_1_FILES pNFS server has the flexibility to revoke individual layouts, and thus fence I/O on a per-file basis. In addition to lease expiration, the reasons a layout can be revoked include: client fails to respond to a CB_LAYOUTRECALL, the metadata server restarts, or administrative intervention. Regardless of the reason, once a client's layout has been revoked, the pNFS server MUST prevent the client from sending I/O for the affected file from and to all data servers; in other words, it MUST fence the client from the affected file on the data servers. Fencing works as follows. As described in Section 13.1, in COMPOUND procedure requests to the data server, the data filehandle provided by the PUTFH operation and the stateid in the READ or WRITE operation are used to ensure that the client has a valid layout for the I/O being performed; if it does not, the I/O is rejected with NFS4ERR_PNFS_NO_LAYOUT. The server can simply check the stateid and, additionally, make the data filehandle stale if the layout specified a data filehandle that is different from the metadata server's filehandle for the file (see the nfl_fh_list description in Section 13.3). Before the metadata server takes any action to revoke layout state given out by a previous instance, it must make sure that all layout state from that previous instance are invalidated at the data servers. This has the following implications. o The metadata server must not restripe a file until it has contacted all of the data servers to invalidate the layouts from the previous instance. o The metadata server must not give out mandatory locks that conflict with layouts from the previous instance without either doing a specific layout invalidation (as it would have to do anyway) or doing a global data server invalidation.
Top   ToC   RFC5661 - Page 334

13.12. Security Considerations for the File Layout Type

The NFSv4.1 file layout type MUST adhere to the security considerations outlined in Section 12.9. NFSv4.1 data servers MUST make all of the required access checks on each READ or WRITE I/O as determined by the NFSv4.1 protocol. If the metadata server would deny a READ or WRITE operation on a file due to its ACL, mode attribute, open access mode, open deny mode, mandatory byte-range lock state, or any other attributes and state, the data server MUST also deny the READ or WRITE operation. This impacts the control protocol and the propagation of state from the metadata server to the data servers; see Section 13.9.2 for more details. The methods for authentication, integrity, and privacy for data servers based on the LAYOUT4_NFSV4_1_FILES layout type are the same as those used by metadata servers. Metadata and data servers use ONC RPC security flavors to authenticate, and SECINFO and SECINFO_NO_NAME to negotiate the security mechanism and services to be used. Thus, when using the LAYOUT4_NFSV4_1_FILES layout type, the impact on the RPC-based security model due to pNFS (as alluded to in Sections 1.7.1 and 1.7.2.2) is zero. For a given file object, a metadata server MAY require different security parameters (secinfo4 value) than the data server. For a given file object with multiple data servers, the secinfo4 value SHOULD be the same across all data servers. If the secinfo4 values across a metadata server and its data servers differ for a specific file, the mapping of the principal to the server's internal user identifier MUST be the same in order for the access-control checks based on ACL, mode, open and deny mode, and mandatory locking to be consistent across on the pNFS server. If an NFSv4.1 implementation supports pNFS and supports NFSv4.1 file layouts, then the implementation MUST support the SECINFO_NO_NAME operation on both the metadata and data servers.


(page 334 continued on part 12)

Next Section