RFC 3986
Uniform Resource Identifier (URI): Generic Syntax
Pages: 61
Internet Standard: 66
→ Errata
Obsoletes: 2732 2396 1808
Updates: 1738
Updated by: 6874 7320 8820
Internet Standard: 66
→ Errata
Obsoletes: 2732 2396 1808
Updates: 1738
Updated by: 6874 7320 8820
Part 1 of 3 – Pages 1 to 16
Network Working Group T. Berners-Lee Request for Comments: 3986 W3C/MIT STD: 66 R. Fielding Updates: 1738 Day Software Obsoletes: 2732, 2396, 1808 L. Masinter Category: Standards Track Adobe Systems January 2005 Uniform Resource Identifier (URI): Generic Syntax Status of This Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2005).Abstract
A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1. Overview of URIs . . . . . . . . . . . . . . . . . . . . 4 1.1.1. Generic Syntax . . . . . . . . . . . . . . . . . 6 1.1.2. Examples . . . . . . . . . . . . . . . . . . . . 7 1.1.3. URI, URL, and URN . . . . . . . . . . . . . . . 7 1.2. Design Considerations . . . . . . . . . . . . . . . . . 8 1.2.1. Transcription . . . . . . . . . . . . . . . . . 8 1.2.2. Separating Identification from Interaction . . . 9 1.2.3. Hierarchical Identifiers . . . . . . . . . . . . 10 1.3. Syntax Notation . . . . . . . . . . . . . . . . . . . . 11 2. Characters . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1. Percent-Encoding . . . . . . . . . . . . . . . . . . . . 12 2.2. Reserved Characters . . . . . . . . . . . . . . . . . . 12 2.3. Unreserved Characters . . . . . . . . . . . . . . . . . 13 2.4. When to Encode or Decode . . . . . . . . . . . . . . . . 14 2.5. Identifying Data . . . . . . . . . . . . . . . . . . . . 14 3. Syntax Components . . . . . . . . . . . . . . . . . . . . . . 16 3.1. Scheme . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2. Authority . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.1. User Information . . . . . . . . . . . . . . . . 18 3.2.2. Host . . . . . . . . . . . . . . . . . . . . . . 18 3.2.3. Port . . . . . . . . . . . . . . . . . . . . . . 22 3.3. Path . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.4. Query . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.5. Fragment . . . . . . . . . . . . . . . . . . . . . . . . 24 4. Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1. URI Reference . . . . . . . . . . . . . . . . . . . . . 25 4.2. Relative Reference . . . . . . . . . . . . . . . . . . . 26 4.3. Absolute URI . . . . . . . . . . . . . . . . . . . . . . 27 4.4. Same-Document Reference . . . . . . . . . . . . . . . . 27 4.5. Suffix Reference . . . . . . . . . . . . . . . . . . . . 27 5. Reference Resolution . . . . . . . . . . . . . . . . . . . . . 28 5.1. Establishing a Base URI . . . . . . . . . . . . . . . . 28 5.1.1. Base URI Embedded in Content . . . . . . . . . . 29 5.1.2. Base URI from the Encapsulating Entity . . . . . 29 5.1.3. Base URI from the Retrieval URI . . . . . . . . 30 5.1.4. Default Base URI . . . . . . . . . . . . . . . . 30 5.2. Relative Resolution . . . . . . . . . . . . . . . . . . 30 5.2.1. Pre-parse the Base URI . . . . . . . . . . . . . 31 5.2.2. Transform References . . . . . . . . . . . . . . 31 5.2.3. Merge Paths . . . . . . . . . . . . . . . . . . 32 5.2.4. Remove Dot Segments . . . . . . . . . . . . . . 33 5.3. Component Recomposition . . . . . . . . . . . . . . . . 35 5.4. Reference Resolution Examples . . . . . . . . . . . . . 35 5.4.1. Normal Examples . . . . . . . . . . . . . . . . 36 5.4.2. Abnormal Examples . . . . . . . . . . . . . . . 36
6. Normalization and Comparison . . . . . . . . . . . . . . . . . 38 6.1. Equivalence . . . . . . . . . . . . . . . . . . . . . . 38 6.2. Comparison Ladder . . . . . . . . . . . . . . . . . . . 39 6.2.1. Simple String Comparison . . . . . . . . . . . . 39 6.2.2. Syntax-Based Normalization . . . . . . . . . . . 40 6.2.3. Scheme-Based Normalization . . . . . . . . . . . 41 6.2.4. Protocol-Based Normalization . . . . . . . . . . 42 7. Security Considerations . . . . . . . . . . . . . . . . . . . 43 7.1. Reliability and Consistency . . . . . . . . . . . . . . 43 7.2. Malicious Construction . . . . . . . . . . . . . . . . . 43 7.3. Back-End Transcoding . . . . . . . . . . . . . . . . . . 44 7.4. Rare IP Address Formats . . . . . . . . . . . . . . . . 45 7.5. Sensitive Information . . . . . . . . . . . . . . . . . 45 7.6. Semantic Attacks . . . . . . . . . . . . . . . . . . . . 45 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 46 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 46 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 46 10.1. Normative References . . . . . . . . . . . . . . . . . . 46 10.2. Informative References . . . . . . . . . . . . . . . . . 47 A. Collected ABNF for URI . . . . . . . . . . . . . . . . . . . . 49 B. Parsing a URI Reference with a Regular Expression . . . . . . 50 C. Delimiting a URI in Context . . . . . . . . . . . . . . . . . 51 D. Changes from RFC 2396 . . . . . . . . . . . . . . . . . . . . 53 D.1. Additions . . . . . . . . . . . . . . . . . . . . . . . 53 D.2. Modifications . . . . . . . . . . . . . . . . . . . . . 53 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 60 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 61
1. Introduction
A Uniform Resource Identifier (URI) provides a simple and extensible means for identifying a resource. This specification of URI syntax and semantics is derived from concepts introduced by the World Wide Web global information initiative, whose use of these identifiers dates from 1990 and is described in "Universal Resource Identifiers in WWW" [RFC1630]. The syntax is designed to meet the recommendations laid out in "Functional Recommendations for Internet Resource Locators" [RFC1736] and "Functional Requirements for Uniform Resource Names" [RFC1737]. This document obsoletes [RFC2396], which merged "Uniform Resource Locators" [RFC1738] and "Relative Uniform Resource Locators" [RFC1808] in order to define a single, generic syntax for all URIs. It obsoletes [RFC2732], which introduced syntax for an IPv6 address. It excludes portions of RFC 1738 that defined the specific syntax of individual URI schemes; those portions will be updated as separate documents. The process for registration of new URI schemes is defined separately by [BCP35]. Advice for designers of new URI schemes can be found in [RFC2718]. All significant changes from RFC 2396 are noted in Appendix D. This specification uses the terms "character" and "coded character set" in accordance with the definitions provided in [BCP19], and "character encoding" in place of what [BCP19] refers to as a "charset".1.1. Overview of URIs
URIs are characterized as follows: Uniform Uniformity provides several benefits. It allows different types of resource identifiers to be used in the same context, even when the mechanisms used to access those resources may differ. It allows uniform semantic interpretation of common syntactic conventions across different types of resource identifiers. It allows introduction of new types of resource identifiers without interfering with the way that existing identifiers are used. It allows the identifiers to be reused in many different contexts, thus permitting new applications or protocols to leverage a pre- existing, large, and widely used set of resource identifiers.
Resource
This specification does not limit the scope of what might be a
resource; rather, the term "resource" is used in a general sense
for whatever might be identified by a URI. Familiar examples
include an electronic document, an image, a source of information
with a consistent purpose (e.g., "today's weather report for Los
Angeles"), a service (e.g., an HTTP-to-SMS gateway), and a
collection of other resources. A resource is not necessarily
accessible via the Internet; e.g., human beings, corporations, and
bound books in a library can also be resources. Likewise,
abstract concepts can be resources, such as the operators and
operands of a mathematical equation, the types of a relationship
(e.g., "parent" or "employee"), or numeric values (e.g., zero,
one, and infinity).
Identifier
An identifier embodies the information required to distinguish
what is being identified from all other things within its scope of
identification. Our use of the terms "identify" and "identifying"
refer to this purpose of distinguishing one resource from all
other resources, regardless of how that purpose is accomplished
(e.g., by name, address, or context). These terms should not be
mistaken as an assumption that an identifier defines or embodies
the identity of what is referenced, though that may be the case
for some identifiers. Nor should it be assumed that a system
using URIs will access the resource identified: in many cases,
URIs are used to denote resources without any intention that they
be accessed. Likewise, the "one" resource identified might not be
singular in nature (e.g., a resource might be a named set or a
mapping that varies over time).
A URI is an identifier consisting of a sequence of characters
matching the syntax rule named <URI> in Section 3. It enables
uniform identification of resources via a separately defined
extensible set of naming schemes (Section 3.1). How that
identification is accomplished, assigned, or enabled is delegated to
each scheme specification.
This specification does not place any limits on the nature of a
resource, the reasons why an application might seek to refer to a
resource, or the kinds of systems that might use URIs for the sake of
identifying resources. This specification does not require that a
URI persists in identifying the same resource over time, though that
is a common goal of all URI schemes. Nevertheless, nothing in this
specification prevents an application from limiting itself to particular types of resources, or to a subset of URIs that maintains characteristics desired by that application. URIs have a global scope and are interpreted consistently regardless of context, though the result of that interpretation may be in relation to the end-user's context. For example, "http://localhost/" has the same interpretation for every user of that reference, even though the network interface corresponding to "localhost" may be different for each end-user: interpretation is independent of access. However, an action made on the basis of that reference will take place in relation to the end-user's context, which implies that an action intended to refer to a globally unique thing must use a URI that distinguishes that resource from all other things. URIs that identify in relation to the end-user's local context should only be used when the context itself is a defining aspect of the resource, such as when an on-line help manual refers to a file on the end- user's file system (e.g., "file:///etc/hosts").1.1.1. Generic Syntax
Each URI begins with a scheme name, as defined in Section 3.1, that refers to a specification for assigning identifiers within that scheme. As such, the URI syntax is a federated and extensible naming system wherein each scheme's specification may further restrict the syntax and semantics of identifiers using that scheme. This specification defines those elements of the URI syntax that are required of all URI schemes or are common to many URI schemes. It thus defines the syntax and semantics needed to implement a scheme- independent parsing mechanism for URI references, by which the scheme-dependent handling of a URI can be postponed until the scheme-dependent semantics are needed. Likewise, protocols and data formats that make use of URI references can refer to this specification as a definition for the range of syntax allowed for all URIs, including those schemes that have yet to be defined. This decouples the evolution of identification schemes from the evolution of protocols, data formats, and implementations that make use of URIs. A parser of the generic URI syntax can parse any URI reference into its major components. Once the scheme is determined, further scheme-specific parsing can be performed on the components. In other words, the URI generic syntax is a superset of the syntax of all URI schemes.
1.1.2. Examples
The following example URIs illustrate several URI schemes and variations in their common syntax components: ftp://ftp.is.co.za/rfc/rfc1808.txt http://www.ietf.org/rfc/rfc2396.txt ldap://[2001:db8::7]/c=GB?objectClass?one mailto:John.Doe@example.com news:comp.infosystems.www.servers.unix tel:+1-816-555-1212 telnet://192.0.2.16:80/ urn:oasis:names:specification:docbook:dtd:xml:4.1.21.1.3. URI, URL, and URN
A URI can be further classified as a locator, a name, or both. The term "Uniform Resource Locator" (URL) refers to the subset of URIs that, in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism (e.g., its network "location"). The term "Uniform Resource Name" (URN) has been used historically to refer to both URIs under the "urn" scheme [RFC2141], which are required to remain globally unique and persistent even when the resource ceases to exist or becomes unavailable, and to any other URI with the properties of a name. An individual scheme does not have to be classified as being just one of "name" or "locator". Instances of URIs from any given scheme may have the characteristics of names or locators or both, often depending on the persistence and care in the assignment of identifiers by the naming authority, rather than on any quality of the scheme. Future specifications and related documentation should use the general term "URI" rather than the more restrictive terms "URL" and "URN" [RFC3305].
1.2. Design Considerations
1.2.1. Transcription
The URI syntax has been designed with global transcription as one of its main considerations. A URI is a sequence of characters from a very limited set: the letters of the basic Latin alphabet, digits, and a few special characters. A URI may be represented in a variety of ways; e.g., ink on paper, pixels on a screen, or a sequence of character encoding octets. The interpretation of a URI depends only on the characters used and not on how those characters are represented in a network protocol. The goal of transcription can be described by a simple scenario. Imagine two colleagues, Sam and Kim, sitting in a pub at an international conference and exchanging research ideas. Sam asks Kim for a location to get more information, so Kim writes the URI for the research site on a napkin. Upon returning home, Sam takes out the napkin and types the URI into a computer, which then retrieves the information to which Kim referred. There are several design considerations revealed by the scenario: o A URI is a sequence of characters that is not always represented as a sequence of octets. o A URI might be transcribed from a non-network source and thus should consist of characters that are most likely able to be entered into a computer, within the constraints imposed by keyboards (and related input devices) across languages and locales. o A URI often has to be remembered by people, and it is easier for people to remember a URI when it consists of meaningful or familiar components. These design considerations are not always in alignment. For example, it is often the case that the most meaningful name for a URI component would require characters that cannot be typed into some systems. The ability to transcribe a resource identifier from one medium to another has been considered more important than having a URI consist of the most meaningful of components. In local or regional contexts and with improving technology, users might benefit from being able to use a wider range of characters; such use is not defined by this specification. Percent-encoded octets (Section 2.1) may be used within a URI to represent characters outside the range of the US-ASCII coded character set if this
representation is allowed by the scheme or by the protocol element in which the URI is referenced. Such a definition should specify the character encoding used to map those characters to octets prior to being percent-encoded for the URI.1.2.2. Separating Identification from Interaction
A common misunderstanding of URIs is that they are only used to refer to accessible resources. The URI itself only provides identification; access to the resource is neither guaranteed nor implied by the presence of a URI. Instead, any operation associated with a URI reference is defined by the protocol element, data format attribute, or natural language text in which it appears. Given a URI, a system may attempt to perform a variety of operations on the resource, as might be characterized by words such as "access", "update", "replace", or "find attributes". Such operations are defined by the protocols that make use of URIs, not by this specification. However, we do use a few general terms for describing common operations on URIs. URI "resolution" is the process of determining an access mechanism and the appropriate parameters necessary to dereference a URI; this resolution may require several iterations. To use that access mechanism to perform an action on the URI's resource is to "dereference" the URI. When URIs are used within information retrieval systems to identify sources of information, the most common form of URI dereference is "retrieval": making use of a URI in order to retrieve a representation of its associated resource. A "representation" is a sequence of octets, along with representation metadata describing those octets, that constitutes a record of the state of the resource at the time when the representation is generated. Retrieval is achieved by a process that might include using the URI as a cache key to check for a locally cached representation, resolution of the URI to determine an appropriate access mechanism (if any), and dereference of the URI for the sake of applying a retrieval operation. Depending on the protocols used to perform the retrieval, additional information might be supplied about the resource (resource metadata) and its relation to other resources. URI references in information retrieval systems are designed to be late-binding: the result of an access is generally determined when it is accessed and may vary over time or due to other aspects of the interaction. These references are created in order to be used in the future: what is being identified is not some specific result that was obtained in the past, but rather some characteristic that is expected to be true for future results. In such cases, the resource referred to by the URI is actually a sameness of characteristics as observed
over time, perhaps elucidated by additional comments or assertions made by the resource provider. Although many URI schemes are named after protocols, this does not imply that use of these URIs will result in access to the resource via the named protocol. URIs are often used simply for the sake of identification. Even when a URI is used to retrieve a representation of a resource, that access might be through gateways, proxies, caches, and name resolution services that are independent of the protocol associated with the scheme name. The resolution of some URIs may require the use of more than one protocol (e.g., both DNS and HTTP are typically used to access an "http" URI's origin server when a representation isn't found in a local cache).1.2.3. Hierarchical Identifiers
The URI syntax is organized hierarchically, with components listed in order of decreasing significance from left to right. For some URI schemes, the visible hierarchy is limited to the scheme itself: everything after the scheme component delimiter (":") is considered opaque to URI processing. Other URI schemes make the hierarchy explicit and visible to generic parsing algorithms. The generic syntax uses the slash ("/"), question mark ("?"), and number sign ("#") characters to delimit components that are significant to the generic parser's hierarchical interpretation of an identifier. In addition to aiding the readability of such identifiers through the consistent use of familiar syntax, this uniform representation of hierarchy across naming schemes allows scheme-independent references to be made relative to that hierarchy. It is often the case that a group or "tree" of documents has been constructed to serve a common purpose, wherein the vast majority of URI references in these documents point to resources within the tree rather than outside it. Similarly, documents located at a particular site are much more likely to refer to other resources at that site than to resources at remote sites. Relative referencing of URIs allows document trees to be partially independent of their location and access scheme. For instance, it is possible for a single set of hypertext documents to be simultaneously accessible and traversable via each of the "file", "http", and "ftp" schemes if the documents refer to each other with relative references. Furthermore, such document trees can be moved, as a whole, without changing any of the relative references. A relative reference (Section 4.2) refers to a resource by describing the difference within a hierarchical name space between the reference context and the target URI. The reference resolution algorithm,
presented in Section 5, defines how such a reference is transformed to the target URI. As relative references can only be used within the context of a hierarchical URI, designers of new URI schemes should use a syntax consistent with the generic syntax's hierarchical components unless there are compelling reasons to forbid relative referencing within that scheme. NOTE: Previous specifications used the terms "partial URI" and "relative URI" to denote a relative reference to a URI. As some readers misunderstood those terms to mean that relative URIs are a subset of URIs rather than a method of referencing URIs, this specification simply refers to them as relative references. All URI references are parsed by generic syntax parsers when used. However, because hierarchical processing has no effect on an absolute URI used in a reference unless it contains one or more dot-segments (complete path segments of "." or "..", as described in Section 3.3), URI scheme specifications can define opaque identifiers by disallowing use of slash characters, question mark characters, and the URIs "scheme:." and "scheme:..".1.3. Syntax Notation
This specification uses the Augmented Backus-Naur Form (ABNF) notation of [RFC2234], including the following core ABNF syntax rules defined by that specification: ALPHA (letters), CR (carriage return), DIGIT (decimal digits), DQUOTE (double quote), HEXDIG (hexadecimal digits), LF (line feed), and SP (space). The complete URI syntax is collected in Appendix A.2. Characters
The URI syntax provides a method of encoding data, presumably for the sake of identifying a resource, as a sequence of characters. The URI characters are, in turn, frequently encoded as octets for transport or presentation. This specification does not mandate any particular character encoding for mapping between URI characters and the octets used to store or transmit those characters. When a URI appears in a protocol element, the character encoding is defined by that protocol; without such a definition, a URI is assumed to be in the same character encoding as the surrounding text. The ABNF notation defines its terminal values to be non-negative integers (codepoints) based on the US-ASCII coded character set [ASCII]. Because a URI is a sequence of characters, we must invert that relation in order to understand the URI syntax. Therefore, the
integer values used by the ABNF must be mapped back to their corresponding characters via US-ASCII in order to complete the syntax rules. A URI is composed from a limited set of characters consisting of digits, letters, and a few graphic symbols. A reserved subset of those characters may be used to delimit syntax components within a URI while the remaining characters, including both the unreserved set and those reserved characters not acting as delimiters, define each component's identifying data.2.1. Percent-Encoding
A percent-encoding mechanism is used to represent a data octet in a component when that octet's corresponding character is outside the allowed set or is being used as a delimiter of, or within, the component. A percent-encoded octet is encoded as a character triplet, consisting of the percent character "%" followed by the two hexadecimal digits representing that octet's numeric value. For example, "%20" is the percent-encoding for the binary octet "00100000" (ABNF: %x20), which in US-ASCII corresponds to the space character (SP). Section 2.4 describes when percent-encoding and decoding is applied. pct-encoded = "%" HEXDIG HEXDIG The uppercase hexadecimal digits 'A' through 'F' are equivalent to the lowercase digits 'a' through 'f', respectively. If two URIs differ only in the case of hexadecimal digits used in percent-encoded octets, they are equivalent. For consistency, URI producers and normalizers should use uppercase hexadecimal digits for all percent- encodings.2.2. Reserved Characters
URIs include components and subcomponents that are delimited by characters in the "reserved" set. These characters are called "reserved" because they may (or may not) be defined as delimiters by the generic syntax, by each scheme-specific syntax, or by the implementation-specific syntax of a URI's dereferencing algorithm. If data for a URI component would conflict with a reserved character's purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
The purpose of reserved characters is to provide a set of delimiting
characters that are distinguishable from other data within a URI.
URIs that differ in the replacement of a reserved character with its
corresponding percent-encoded octet are not equivalent. Percent-
encoding a reserved character, or decoding a percent-encoded octet
that corresponds to a reserved character, will change how the URI is
interpreted by most applications. Thus, characters in the reserved
set are protected from normalization and are therefore safe to be
used by scheme-specific and producer-specific algorithms for
delimiting data subcomponents within a URI.
A subset of the reserved characters (gen-delims) is used as
delimiters of the generic URI components described in Section 3. A
component's ABNF syntax rule will not use the reserved or gen-delims
rule names directly; instead, each syntax rule lists the characters
allowed within that component (i.e., not delimiting it), and any of
those characters that are also in the reserved set are "reserved" for
use as subcomponent delimiters within the component. Only the most
common subcomponents are defined by this specification; other
subcomponents may be defined by a URI scheme's specification, or by
the implementation-specific syntax of a URI's dereferencing
algorithm, provided that such subcomponents are delimited by
characters in the reserved set allowed within that component.
URI producing applications should percent-encode data octets that
correspond to characters in the reserved set unless these characters
are specifically allowed by the URI scheme to represent data in that
component. If a reserved character is found in a URI component and
no delimiting role is known for that character, then it must be
interpreted as representing the data octet corresponding to that
character's encoding in US-ASCII.
2.3. Unreserved Characters
Characters that are allowed in a URI but do not have a reserved
purpose are called unreserved. These include uppercase and lowercase
letters, decimal digits, hyphen, period, underscore, and tilde.
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
URIs that differ in the replacement of an unreserved character with its corresponding percent-encoded US-ASCII octet are equivalent: they identify the same resource. However, URI comparison implementations do not always perform normalization prior to comparison (see Section 6). For consistency, percent-encoded octets in the ranges of ALPHA (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E), underscore (%5F), or tilde (%7E) should not be created by URI producers and, when found in a URI, should be decoded to their corresponding unreserved characters by URI normalizers.2.4. When to Encode or Decode
Under normal circumstances, the only time when octets within a URI are percent-encoded is during the process of producing the URI from its component parts. This is when an implementation determines which of the reserved characters are to be used as subcomponent delimiters and which can be safely used as data. Once produced, a URI is always in its percent-encoded form. When a URI is dereferenced, the components and subcomponents significant to the scheme-specific dereferencing process (if any) must be parsed and separated before the percent-encoded octets within those components can be safely decoded, as otherwise the data may be mistaken for component delimiters. The only exception is for percent-encoded octets corresponding to characters in the unreserved set, which can be decoded at any time. For example, the octet corresponding to the tilde ("~") character is often encoded as "%7E" by older URI processing implementations; the "%7E" can be replaced by "~" without changing its interpretation. Because the percent ("%") character serves as the indicator for percent-encoded octets, it must be percent-encoded as "%25" for that octet to be used as data within a URI. Implementations must not percent-encode or decode the same string more than once, as decoding an already decoded string might lead to misinterpreting a percent data octet as the beginning of a percent-encoding, or vice versa in the case of percent-encoding an already percent-encoded string.2.5. Identifying Data
URI characters provide identifying data for each of the URI components, serving as an external interface for identification between systems. Although the presence and nature of the URI production interface is hidden from clients that use its URIs (and is thus beyond the scope of the interoperability requirements defined by this specification), it is a frequent source of confusion and errors in the interpretation of URI character issues. Implementers have to be aware that there are multiple character encodings involved in the
production and transmission of URIs: local name and data encoding, public interface encoding, URI character encoding, data format encoding, and protocol encoding. Local names, such as file system names, are stored with a local character encoding. URI producing applications (e.g., origin servers) will typically use the local encoding as the basis for producing meaningful names. The URI producer will transform the local encoding to one that is suitable for a public interface and then transform the public interface encoding into the restricted set of URI characters (reserved, unreserved, and percent-encodings). Those characters are, in turn, encoded as octets to be used as a reference within a data format (e.g., a document charset), and such data formats are often subsequently encoded for transmission over Internet protocols. For most systems, an unreserved character appearing within a URI component is interpreted as representing the data octet corresponding to that character's encoding in US-ASCII. Consumers of URIs assume that the letter "X" corresponds to the octet "01011000", and even when that assumption is incorrect, there is no harm in making it. A system that internally provides identifiers in the form of a different character encoding, such as EBCDIC, will generally perform character translation of textual identifiers to UTF-8 [STD63] (or some other superset of the US-ASCII character encoding) at an internal interface, thereby providing more meaningful identifiers than those resulting from simply percent-encoding the original octets. For example, consider an information service that provides data, stored locally using an EBCDIC-based file system, to clients on the Internet through an HTTP server. When an author creates a file with the name "Laguna Beach" on that file system, the "http" URI corresponding to that resource is expected to contain the meaningful string "Laguna%20Beach". If, however, that server produces URIs by using an overly simplistic raw octet mapping, then the result would be a URI containing "%D3%81%87%A4%95%81@%C2%85%81%83%88". An internal transcoding interface fixes this problem by transcoding the local name to a superset of US-ASCII prior to producing the URI. Naturally, proper interpretation of an incoming URI on such an interface requires that percent-encoded octets be decoded (e.g., "%20" to SP) before the reverse transcoding is applied to obtain the local name. In some cases, the internal interface between a URI component and the identifying data that it has been crafted to represent is much less direct than a character encoding translation. For example, portions of a URI might reflect a query on non-ASCII data, or numeric
coordinates on a map. Likewise, a URI scheme may define components with additional encoding requirements that are applied prior to forming the component and producing the URI. When a new URI scheme defines a component that represents textual data consisting of characters from the Universal Character Set [UCS], the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then only those octets that do not correspond to characters in the unreserved set should be percent- encoded. For example, the character A would be represented as "A", the character LATIN CAPITAL LETTER A WITH GRAVE would be represented as "%C3%80", and the character KATAKANA LETTER A would be represented as "%E3%82%A2".
(page 16 continued on part 2)