Network Working Group                                         A. Bhushan
Request for Comments: 310                                        MIT-MAC
NIC: 9261                                                  April 3, 1972


            Another Look At Data And File Transfer Protocols
	
   Our experience with ad hoc techniques of data and file transfer over
   the ARPANET together with a better knowledge of terminal IMP (TIP)
   capabilities and Datacomputer requirements has indicated to us that
   the Data Transfer Protocol (DTP) (see ref 1) and the File Transfer
   Protocol (FTP) (see ref 2) could undergo revision.  Our effort in
   implementing DTP and FTP has revealed areas in which the protocols
   could be simplified without degrading their usefulness.

   This paper suggests some specific changes in DTP and FTP that should
   make them more useful and/or simplify implementation.  The attempt
   here is to stimulate thinking so that we may come up with a better
   protocol at the forthcoming Data and File Transfer Workshop (see ref
   3).

Experience to Date

   A number of ad hoc techniques of transmitting data and files across
   the ARPANET already exist.  Perhaps, the most versatile of these
   existing methods is the TENEX "CPYNET" system.  The "CPYNET" system
   uses an ad hoc or interim file transfer protocol developed by Ray
   Tomlinson and others at BBN to transmit files among the TENEX systems
   on the ARPANET. [Private Communication with Bill Crowther, BBN.]

   In CPYNET, the using process goes through the Initial Connection
   Protocol (ICP) to server socket 7, establishing a full-duplex
   connection with an 8-bit byte size.  Control information, including
   user name, password, command (read, write, or append), file name, and
   byte size for the data connection is transmitted from the using
   process to the serving process.  The original full-duplex connection
   is then closed, and a new full-duplex connection is established using
   the original socket numbers but with possibly a different byte size.
   The file is now transmitted on this newly established connection.
   The end-of-file is indicated by closing the connection (the mode of
   transfer is thus similar to DTP "indefinite bit-stream").

   CPYNET has been used quite extensively for transfer of TENEX system
   files.  Because data is not reformatted, and because the optimum
   connection byte size may be used for data transfer, CPYNET is quite
   efficient.  The PDP-10 (and there are quite a lot in the ARPANET)
   works more efficiently with a 36 bit byte size which minimizes
   packing and unpacking of data, and increases effective I/O speed

   (bit rate is 36 times the I/O word transfer rate instead of 8 times).
   The closing and reopening of connections does increase overhead but
   this is small in TENEX when compared with inefficiency introduced in
   data transfer using an inappropriate byte size.

   Data and file transfer has been achieved at other sites by a simple
   modification of the user TELNET to enable the transfer of ASCII files
   as terminal I/O data streams within the constraints of the TELNET
   protocol.  An example of this approach is the use of the "send.file"
   and "script" features within the MIT-DMCG user-TELNET.  Send.file
   enables the PDP-10 (DMCG) user to transmit his local ASCII files to a
   receiving process such as an editor at the remote host via a TELNET
   connection.  The program allows for a variable buffer size for
   transmission, and measures the transfer rate of information bits.
   Script enables a user to receive an ASCII file from a remote host by
   essentially printing it out (the terminal output stream is directed
   to a local file).

   Our initial experience with the use of send.file program has affirmed
   the almost linear relationship between buffer size and transmission
   rate (inverse relationship to processing cost) until the limits
   imposed by allocates, NCP sending buffers, the IMP message size, or
   the receiving process speed, are reached.  Our experiments have
   indicated that TELNET processes in which the receiving process
   "looks" at each character are slow and expensive.  The transfer rate
   to most TELNET receiving processes ranges between 200 and 2,000 bits
   per second.  The NCP-to-NCP transmission rate however is an order-
   of-magnitude higher (2,000 to 20,000 bits per second).

   A variation of the above method which avoids the character-by-
   character processing of TELNET, is transmitting files via auxiliary
   connections (other than the TELNET connections) with or without the
   use of DTP.  We are collecting data on response times, connect times
   and transfer speeds employing different transfer and buffering
   strategies.

TIP Capabilities and TIP Users

   It appears now that TIPs will not support DTP in its present form.
   The more elaborate TIPs with magnetic tape units will however,
   support the DTP block mode (descriptor and counts) [Private
   Communication with Bill Crowther, BBN.]  It is inconvenient, at the
   very least, for a TIP user to use services based on DTP (such as
   remote job service, file transfer, mail, and Datacomputer).  The TIP
   philosophy is that "the computational load and storage should be in
   the hosts or in the terminals and not in the terminal processor."
   (See ref 4.) To be consistent with this philosophy the protocols
   should be simple and convenient to use from the user viewpoint.

   Ideally, TIP users would like to connect (using the initial
   connection protocol) to the advertised service socket (including
   logger socket1) in the remote host and type their commands in a
   uniform, easy to use, format.  Allowing the use of ASCII within DTP
   would facilitate this.  (An alternate approach is extending TELNET to
   include DTP modes, particularly the indefinite bit-stream mode.)
   Another step would be to use printable ASCII strings instead of
   numeric codes for commands and arguments in user-level protocols.
   Use of standard file system commands (with uniform interpretation and
   format) will lead towards the existence of a Network Virtual File
   System, much in the same line as Network Virtual Terminal defined in
   TELNET protocol.

   The transparent mode in DTP was specifically included to allow
   convenient use by TIPs.  Since the TIPs will not support transparent
   mode, it makes sense to do away with it.  This change would lead to a
   simplier DTP which allows transfer in Block mode, and the indefinite
   bit-stream mode.  (The suggested default which would be acceptable to
   all including the TIPs, as it involves no overhead.).  We can then
   make optional or do away with the now mandatory modes available
   handshake.  The using process can indicate if it also accepts the
   block mode for transfer.  (Either by modes available transaction, or
   by an argument in the command string).  The server should accept
   input in DTP mode as well as ASCII.  These fundamental changes in DTP
   will make communication with TIPs a lot easier.

   TIP users who do not have a mediating user-FTP process and a file
   system in their TIP, would probably want to transfer files from input
   devices or to output devices such as line printer, card reader or
   punch, or magnetic tape.  These devices "listen" on specific "ports"
   or sockets on a TIP.  It would be desirable to modify FTP to allow
   sending data to a specified socket in a specified mode and type.  TIP
   users would then find it convenient to obtain listing of their files
   on a high-speed line printer, input their files from a card reader,
   and keep back-up on cards or magnetic tapes.

Datacomputer Requirements

   We have been having a continuing dialogue with CCA personnel (Dick
   Winter in particular), regarding CCA's plans for data and file
   transfer on the Datacomputer, and their specific requirements.  Dick

   Winter will be speaking on this subject at the Data and File Transfer
   Workshop.  This is an attempt to summarize the main points of our
   discussion, and their implication for data and file transfer.

   First, CCA appears quite flexible at this stage regarding the manner
   in which Datacomputer is to be used.  Even the Datalanguage (see ref
   5) is flexible and can undergo change.  However, CCA would like some
   changes in the current file transfer protocol and its envisioned use.

   Ideally, CCA would like to see a single full-duplex connection for
   transfer of all control information which is in Datalanguage.  This
   information is generated by a process, which may be a user at a
   console, or a user program.  Ability to inter-mix data and control
   information would be definite advantage.  The Datacomputer would
   probably support DTP and allow use of TELNET-ASCII.

   Data may alternatively be sent to or received from a separate user
   defined port (which may be a socket).  It would be advantageous if a
   user could instruct the Datacomputer to transfer data to or from a
   file in remote system via FTP (assuming a server-FTP in remote
   system).  CCA is currently not committed to this idea, but is
   considering it.

   In the CCA view, the Datacomputer represents a data management
   facility with Datalanguage as its command language.  From the
   viewpoint of Datacomputer as an FTP server, FTP commands be a subset
   of the Datalanguage.  It is therefore desirable that FTP commands be
   printable ASCII strings instead of numeric codes.

Remote Job Service Requirements

   Initially two separate protocols were proposed for Remote Job Service
   (RJS).  One was the NETRJS protocol (see ref 6) for remote job
   service from large Hosts and the other was the NETRJT Protocol (see
   ref 7) for remote job service from TIPs (and other mini-Hosts).  The
   current thinking however, is to move towards a single RJS with "as
   much overlap as possible between the methods of dealing with these
   two user populations."  (See ref 8.)  Perhaps inclusion of ASCII
   within DTP would make this feasible.

   The existing proposals for DTP and FTP have been considered somewhat
   less than optimal for RJS needs.  Specific drawbacks of DTP and FTP
   have been pointed out in the handling of data structures and data
   types.  Most of these problems seem relatively easy to resolve.  It
   would involve making Network ASCII the default data type (acceptable
   to all hosts) and providing a way in FTP for proposing and rejecting
   alternative data types and data structures.

   Another inadequacy of FTP (which pertains to other applications as
   well) is in the area of error recovery.  Currently there is no way to
   "restart" transmission if an element in the transmission path fails.
   One solution suggested has involved the use of sequence number (see
   ref 9).  A number of other solutions exist to the problem.  These are
   discussed later in the section 'FTP Reconsidered'.

DTP Reconsidered

   The aspiration for DTP was that it would provide a uniform mechanism
   for separating information into its logical structure (records,
   files, and control), and rudimentary error control.  The evaluation
   of DTP and its modes should be on the basis of speed (real-time),
   efficiency (processing cost), reliability (error control and
   recovery), and the ease of its use.

   It is now clear that unless DTP was significantly revised, the TIP
   and other mini-Host user would find it difficult to use services
   based on use of DTP.  Allowing the use of ASCII within DTP, and using
   defaults instead of the "modes available" handshake, would alleviate
   this problem, but compromise the DTP error control function.  Whether
   error control belongs at the DTP level or at a higher level needs
   further discussion.

   DTP, in its present form, does not provide sufficient error control
   and recovery procedures.  To make DTP more useful, either it should
   be simplified (at least from a user viewpoint), or it should be
   extended to include better error control with built in error
   recovery, and possible handling of data types and data structures.

   In the simplified version, DTP would only be a format procedure in
   which data could be transmitted as ASCII (no format) with escape to
   an 8-bit transparent (indefinite bit-stream) mode or in data blocks
   (descriptor and count mode).  The choice of which mode to use, and
   all error control, error recovery, and aborts would be handled by the
   higher-level protocol.

   The utility of the block mode in data transfer has been questioned by
   many who suggest that it puts a large overhead for providing the
   simple function of indicating end-of-file, and separating data and
   control information.  The alternative data transfer strategy is to
   use separate connections for control and data information and/or
   close and reopen connections.  This causes an overhead of a different
   sort, but has the advantage that the byte size for connection may be
   chosen to optimize data transfer.

   A drawback of present DTP is that it is geared to transfer of 8-bit
   bytes.  Perhaps a good strategy for data transfer would be to allow
   sending data in an agreed upon transfer mode.  The transfer mode
   chosen should determine the byte size for connection, the data type
   chosen, and any data structure information.  This mode may be chosen
   at the DTP level, or at the using protocol level.

FTP Reconsidered

   The aspiration for FTP was that it would facilitate file management
   and file transfer in the ARPANET Virtual File System.  FTP success
   should be evaluated by the extent of its use, convenience and
   efficiency in its use, and its suitability for other applications
   such as Datacomputer, RJS, and Mail.

   Wide use of FTP would be possible if a user could use an FTP-server
   directly without the help of a mediating DTP/FTP-User process.  This
   would require that commands be ASCII strings instead of numeric
   codes, and that ASCII characters be an acceptable input.  Such a
   change in FTP would greatly increase its acceptance at the cost of
   making the server-implementation more complex.  Combined
   implementation, however, would be simplified as the mediating FTP-
   user process (if used at all) would be simplified.

   Efficiency of transfer is an important factor affecting the
   usefulness of FTP.  File transfer may be very expensive (in terms of
   CPU time) and slow (in real-time) if an inappropriate transfer
   strategy is used (e.g., inappropriate byte size).  Every attempt
   should be made to optimize transfer of data.  A good strategy may be
   to allow transfer of files over a separate connection or close and
   reopen connections (using perhaps a different byte size).  A problem
   with indicating end-of-file by closing connection is that is
   uncertain if the connection was closed because end-of-file was
   reached, or because of a failure or error condition.  Perhaps "NCP
   interrupts" could be used in addition to a "close" to indicate
   definite end-of-file condition.

   A drawback in the present FTP strategy is that it has no restart
   procedure.  One proposal for restart has involved the use of the
   sequence numbers used in DTP block mode.  Our feeling is that perhaps
   restart may best be accomplished by incorporating a command in FTP
   that allows a user to specify the place in file where to begin
   retransmission.  A possible solution is to use the "SPF" command
   implemented in the UCSB Simple-Minded File System (see ref 10).
   Another solution may be to have optional arguments for retrieve and
   store commands that allow selective retrieval and replacement
   (specified by bits, character, words, lines, pages or segments).

   Another useful addition to FTP would be a protocol procedure between
   user and server to agree to data type, data structure, and mode for
   file transfer.  This would enable the user and server to reach the
   optimum file transfer strategy acceptable to both.

Concluding Remarks

   We have discussed in this paper what we see as the major problem
   areas in the present DTP and FTP specifications.  We hope this
   discussion will stimulate thinking, so that we can arrive at revised
   specifications for DTP and FTP that satisfy all the diverse needs in
   an elegant manner.

REFERENCES

      1. The Data Transfer Protocol, Bhushan, et al, NWG/RFC #264, NIC
   #7212.

      2. The File Transfer Protocol, Bhushan, et al, NWG/RFC #265, NIC
   #7213.

      3. Data and File Transfer Workshop Announcement, A. Bhushan,
   NWG/RFC #309, NIC #9260.

      4. The Terminal IMP for the ARPA Compuer Network, Ornstein, et al,
   SJCC, 1972, NIC #8218.

      5. Datalanguage, Computer Operation of America, Datacomputer
   Project, Working Paper No.3, October 29, 1971, NIC #8208.

      6. Interim NETRJS Specifications, R. T. Braden, NWG/RFC #189, NIC
   #7133.

      7. NETRJT - - Remote Job Service Protocol for TIPs, R. T. Braden,
   NWG/RFC #283, NIC #8165.

      8. RJS Protocol Meeting Notes, 25 February 1972, A. McKenzie
   (limited distribution).

      9. A Suggested Addition to File Transfer Protocol, A. McKenzie,
   NWG/RFC #281, NIC #8163.

      10. Network Specifications for UCSB's Simple-Minded Files System,
   James E. White, NWG/RFC #122, NIC #5834
	
        [This RFC was put into machine readable form for entry]
     [into the online RFC archives by Hélène Morin, Viagénie 10/99]