You Pressed Enter. Now What?

Pressing Enter does not immediately send an HTTP request. Before the browser can do that, it needs to interpret what you meant by the text you typed in the address bar. Is it a complete URL, an incomplete URL, a search query, or an alias to a bookmark?

Once the browser has determined the full URL of the resource you want to load, it still has several decisions to make: where to send the request, how to construct it, whether the request is permitted by the browser’s security rules, and whether a valid response exists in the local cache. Everything before sending a single byte over the network.

You Pressed Enter, But What Did You Mean?

In modern browsers, the address bar accepts many kinds of input: complete URLs, partially formed URLs, search terms, bookmark names, aliases to sites, and, in some browsers, bangs. To help you choose among them, the browser gives you multiple suggestions drawn from your browsing history, bookmarks, search completion, and other sources.

The decision of which path to take is based on many aspects of the text you have typed:

• Does the text follow the structure of a URL?
• Is the last part of the domain a valid top-level domain?
• Does it contain spaces?
• Does it match a bookmark name?

For example, if you type Example Search in the address bar, the browser may interpret it as a search query. It then needs to look up your default search engine and how to expand this query to the full URL. With Google, the browser would expand it to be https://www.google.com/search?q=Example+Search, creating a full URL for your search. This works similarly for bangs: a prefix determines which service to use and how to expand the remaining text into a full URL.

Another way browsers try to be clever on how to determine which resource to load is by using your bookmark names and history data. Consider that you have typed only a single word in the address bar, it can be a domain that doesn’t have a ., like localhost, but it can also be a site name that you have in your bookmarks or in the history. In these cases the browser can expand the name to be this entry’s full URL.

Turning Text Into a URL

After interpreting and expanding what you typed, the browser eventually arrives at a URL. But the URL is more than just a string of text: it has a defined structure that the browser needs to split into individual parts before using it.

RFC 3986 defines the generic syntax of URIs and the components they can contain. Modern browsers, however, implement URL parsing according to the WHATWG URL Standard, which defines how web URLs are parsed and serialized.

A URL can contain the following components:

• Scheme: identifies how the resource should be accessed, such as http or https
• User information: optional information associated with the authority, historically used for credentials (we’ll see how attackers abuse this later in this article)
• Host: identifies the server, usually through a domain name or IP address
• Port: optionally identifies the specific network port to connect to and, if omitted, the scheme usually implies a default
• Path: identifies the resource being requested
• Query: provides additional parameters or information to the server
• Fragment: identifies a specific part of the resource, handled client-side and not sent to the server

As an example, here is a URL containing all of these components:

Many programming languages provide standard libraries for parsing URLs. For example, in JavaScript, both Node.js and the browser expose the URL class:

const url = new URL(
  "https://user:password@example.com:8080/products/shoes?color=blue&size=10#reviews",
);

console.log({
  protocol: url.protocol,
  username: url.username,
  password: url.password,
  hostname: url.hostname,
  port: url.port,
  pathname: url.pathname,
  search: url.search, // This is the query
  hash: url.hash, // This is the fragment
});

But not every component needs to be explicitly present. For example, typing these two strings into the address bar can lead to the same request for this blog article:

This works because the browser can fill in information that wasn’t explicitly provided. In this case, it determines that HTTPS should be used by first guessing the scheme and then checking the internal policies for this site’s security (more on this shortly). Browsers base this guess on their internal HTTPS-first policy, most modern ones try using HTTPS first and, if it fails, fall back to plain HTTP. The other missing piece is the port, and since HTTPS uses port 443 by default, the browser will use this port to connect to the server.

URL Normalization

The equivalence between two URLs can’t be checked with a simple string comparison, because some parts of a URL can accept certain sequences of characters that are different strings but, in the end, have the same normalized representation. To address this, RFC 3986 defines several techniques that can be used when comparing the strings.

Considering these two URLs as examples, http://a/b/c/%7Bfoo%7D and hTTP://A:80/./b/../b/%63/%7bfoo%7d, let’s apply some of the techniques described in the RFC.

The first technique defined is case normalization. Since the scheme and the host are case-insensitive, the RFC defines they should be normalized to be lowercase. The RFC also defines that hexadecimal digits A-F inside percent encoding should be uppercase. So our example URLs become http://a/b/c/%7Bfoo%7D and http://a:80/./b/../b/%63/%7Bfoo%7D.

Another technique is the percent-encoding normalization. Percent-encoding represents a byte as % followed by two hexadecimal digits. For example, % is encoded as %25, while a space can be represented as %20. During normalization, percent-encoded bytes representing unreserved characters can be decoded. In our example, %63 becomes c, while %7B remains encoded because { is not an unreserved character. Applying this to our URLs results in http://a/b/c/%7Bfoo%7D and http://a:80/./b/../b/c/%7Bfoo%7D.

Another technique defined in the RFC is path segment normalization. URL paths can also contain the special segments . and .., much like filesystem paths. A . segment refers to the current level and can be removed, while .. removes itself together with the preceding path segment. Doing this, our example URLs become http://a/b/c/%7Bfoo%7D and http://a:80/b/c/%7Bfoo%7D.

Each scheme can define its own normalization steps to be applied. In our examples we are using HTTP URLs, so some of the HTTP normalization rules apply. For HTTP, port 80 is the default. Because the second URL explicitly specifies the scheme’s default port, the port can be omitted during normalization. So now our URLs are http://a/b/c/%7Bfoo%7D and http://a/b/c/%7Bfoo%7D.

So we started with two very different strings, but after applying the normalization techniques, both have the same normalized form http://a/b/c/%7Bfoo%7D.

These RFC normalization techniques are useful for understanding why different URI strings can compare as equivalent, but browsers do not simply run this exact sequence of transformations. For actual browser URL parsing and serialization behavior, the WHATWG URL Standard defines the relevant algorithms. You can check your browser’s implementation of the algorithm by running the code below in the developer tools console.

const url = new URL("hTTP://A:80/./b/../b/%63/%7bfoo%7d");

console.log(url.href);

You will see that the output is different from our manual normalization, mainly because of the percent-encoding normalization, where the browser won’t transform the %63 into c. This happens because browsers don’t normally decode the percent-encoded unreserved characters, assuming that, if they were written this way, the developer meant to have them like this. Some important APIs require cryptographically signed requests, and decoding the path could invalidate that signature. Two big examples of this are AWS APIs and OAuth.

URI vs URL

You may have noticed that I have used the terms URI and URL interchangeably during this article and also that the RFC is focused on defining URIs, not URLs. This is a historic heritage from the early stages of the definitions of the patterns that govern the modern internet. URLs are a subset of URIs.

URIs are Uniform Resource Identifiers, used to identify a resource, be it a physical or digital one. You may have seen many examples of URIs in the wild:

• https://google.com - Google’s website
• mailto:john@example.com - mail links on websites follow a different scheme
• urn:li:activity:7488389617277263872 - LinkedIn resources are identified by URNs

A URL is a Uniform Resource Locator, so, besides identifying the resource, it carries information on how to locate it: protocol, server, port, path.

Origin

Browsers use origins to determine whether resources belong to the same security context, which is important for policies such as the same-origin policy and CORS (we’ll cover this in a later article on browser security).

An origin is the combination of a URL’s scheme, host, and port. Two URLs are considered to have the same origin only if all these components match.

Consider the URLs below. URLs 1 and 2 have the same origin because the default port for HTTPS is 443. URL 3 has a different scheme, URL 4 has a different host, and URL 5 has a different port.

1. https://example.com
2. https://example.com:443
3. http://example.com
4. https://api.example.com
5. https://example.com:8443

JavaScript’s URL class also exposes the parsed URL’s origin:

const url = new URL("https://example.com:443/path");

console.log(url.origin);
// https://example.com

Before Going to the Network

Now the browser knows what resource you want and how to request it. But that doesn’t mean that it will fire a request. Before even opening a network connection to the server, the browser has to check if the request passes on its security policies and even if it can handle this request locally.

HSTS

The HTTP Strict Transport Security (HSTS) is a browser policy that says that the browser should treat this site as HTTPS only, with no exceptions. This policy is based on the Strict-Transport-Security header (we’ll cover HTTP headers in a later article on HTTP), where the server informs the browser for how long it should maintain this policy for the domain.

This header can contain three pieces of information:

• max-age: for how many seconds the browser should enforce this policy, required
• includeSubDomains: to identify if this policy should be applied to all of the subdomains
• preload: to identify for browser maintainers that you are ready to be included in the preloaded HSTS list

When a browser receives an HTTPS response with the Strict-Transport-Security header, it updates the internal list with the new expiration date and subdomains flag.

Browsers already ship with a preloaded HSTS list containing domains that should always be accessed over HTTPS. Some top-level domains, such as .dev, are preloaded as a whole, so every site under them is treated as HTTPS-only (that’s why our example before works). This preloaded list solves HSTS’s first-visit problem. If the browser does not yet know that a site uses HSTS, it may initially send an HTTP request, receive an HTTPS redirect, and only then learn the HSTS policy from the HTTPS response.

This policy differs from a redirect because there is no HTTP request to the server with a redirect response, enforcing that the browser should change the scheme to use HTTPS even before this first request is fired.

In Chrome you can use chrome://net-internals/#hsts to check for Chrome’s internal HSTS policies. To check if your website is ready to be preloaded you can use Google’s HSTS Preload Service. And using curl you can check for the response headers of a website to validate the HSTS header. GitHub is a good example of using this header.

curl -I https://github.com

Service Workers

Most modern web apps provide some background data synchronization or even offline usage. This is made possible by using the ServiceWorker API provided by modern browsers.

Service workers act as a middle man between the application and the network request. For a worker to start processing the events of the page, first, the application needs to register it, providing a scope. The scope of a service worker is a combination of the application origin and a path. The path is used to determine for which request the browser will execute the worker. You define the base path, so all requests that are nested under this path will have the worker executed.

The code below shows an example of the call to register a worker from the script sw.js in all the paths that are nested under /app.

navigator.serviceWorker.register("sw.js", {
  scope: "/app",
});

Workers can react to events triggered by the application, and one of the most commonly used is the fetch event. Every request triggered by the application that is under the scope of a worker will pass through this event handler. This allows the service worker to intercept and rewrite all the requests based on the logic defined by the application developer.

In the following example the request is intercepted and, if a cached response is available, the worker returns the cached data, and if not, it goes to the network, fetches the data and adds it to the cache. This works as a cache for when the browser is online and as a simple offline cache for when the network has been disconnected.

self.addEventListener("fetch", (event) => {
  event.respondWith(
    caches.match(event.request).then((cachedResponse) => {
      // If we have a cached response, use it and skip the network entirely.
      if (cachedResponse) {
        return cachedResponse;
      }

      // Otherwise, go to the network and store a copy for next time.
      return fetch(event.request).then((networkResponse) => {
        return caches.open("v1").then((cache) => {
          cache.put(event.request, networkResponse.clone());
          return networkResponse;
        });
      });
    }),
  );
});

Registering a service worker doesn’t hand it control immediately. The page that triggered the registration keeps loading exactly as it would without one, including whatever it requests while loading, which still goes straight to the network. And even once the worker finishes activating, it won’t intercept requests from that same page until the next navigation. Only a reload or new tab puts it in charge, unless the worker calls clients.claim() in its activate handler to take over already-open pages right away.

Since the worker can intercept any request under its scope, it can be a vector for man-in-the-middle attacks, where attackers put themselves in between the application and the server to steal and manipulate the data. To mitigate this, browsers only allow service worker registrations from secure contexts where the identity of the source can be trusted, be it via TLS certificates (the base for HTTPS) or local development contexts.

Browser Cache

The browser cache is a different cache from the service workers one. This one is managed by the browser based on the cache control headers returned by the server (we’ll cover this in a later article on caching and rendering).

When an application requests some data, the browser checks in the cache to see if there is a version of the resource available to return without requesting it again. This works automatically, without any intervention from the user or the developer and helps to speed up the loading of the page.

As an example, consider the logo of a website, present at the top of the page. This file won’t change frequently, so the browser can keep a local copy of the image for a long time, defined by the server in the cache headers, and save a network request on subsequent page loads.

Browser Extensions

Modern browsers already provide a bunch of features embedded on them, but they also provide ways for developers to add other features via the extensions API. With these APIs developers can integrate their password managers with the autocomplete options of the browser, provide customization features, and, most notably, provide ad-blocking functionality.

Since browser extensions can request permissions to read and manipulate data, and look at network requests in all sites you access during your browsing sessions, you should be cautious about which extensions you install and which sites they are allowed to access. Browsers’ extension stores already vet extensions they provide for basic security risks, but looking at the source of the extension, reviews and install base is a basic action before installing a new extension.

Which Network Path Should Be Used?

When a browser checks the service workers and local caches and determines that a network request will be needed, typically, it sends the request directly to the application server. But some network setups require the user to use a network proxy, be it on a browser or system level, that does the request for the user, providing the functionality of enforcing company security policies or an in-network cache, among other uses.

When a proxy is used, the browser requests the resource from the proxy instead of the destination server. Then, the proxy opens its own connection to the application server and, once it gets the response, forwards it back to the browser. This flow is called a forward proxy, since the proxy forwards your request to the application server, differing from a reverse proxy, which we’ll cover in a later article on proxies, CDNs, and load balancers.

Another option for network administrators is using Proxy Auto-Configuration files, known as PAC files. These files are used by browsers to determine, on a request-by-request basis, if they should use a proxy, allowing more complex decisions on how to use proxies. The path to the file is manually configured or an auto-discovery service is used via DHCP or with a well-known DNS name. This auto-configuration is known as Web Proxy Auto-Discovery (WPAD).

A PAC file is a JavaScript file that exposes a FindProxyForURL function that receives the URL (most browsers strip the path and query components for HTTPS requests to protect the user privacy) and hostname of the server being accessed and returns a string describing the proxy configuration to be used. An example of a PAC file is shown below.

function FindProxyForURL(url, host) {
  if (dnsDomainIs(host, ".internal.company.com")) {
    return "DIRECT";
  }

  return "PROXY proxy.company.com:8080; PROXY backup-proxy.company.com:8080";
}

In this example, if a request is for an internal domain, the connection will be direct to the application, but, in any other case, the browser will use the proxy configuration. For this configuration there is a main proxy proxy.company.com:8080 and, if the browser can’t connect to this server, it’ll try the backup proxy backup-proxy.company.com:8080.

The example shows only one of the possible use cases, checking only if the host matches the internal company domains, while the PAC file specification provides multiple functions that allow you to define conditions based on DNS queries and time, among others. If you want more information on the available functions and the return format for PAC files, or even more examples on use cases, check the MDN page on this topic.

With the proxy configuration chosen, the browser can proceed to the next steps of the request, knowing which party will be responsible for each subsequent task, the browser or the proxy. If a DIRECT configuration is received, the browser is responsible for the name resolution and the subsequent requests, and if a proxy config is received, the browser forfeits some of the control of the requests to the proxy server.

Does the Browser Already Know Where to Go?

Now that the browser knows what resource you are trying to get and how to reach the internet to request it, the next step is knowing where to go.

Internet communications run on top of the Internet Protocol (IP), be it IPv4 or IPv6, but until now all we’ve dealt with are URLs. How does the browser get IP addresses? This is the work of the DNS subsystem, but the browser and the OS can help.

For every domain you access, the browser and the OS each keep their own list of all the domains already translated to IPs so they can speed up the process for subsequent requests. But if there is no cached result for the domain you are trying to access, then the request to translate the domain into an IP address is sent to the DNS subsystem in your OS.

We’ll cover all of this in the next article.

The Browser Still Has More Decisions Ahead

Even though all the basic information needed to initiate the request for the resource has been acquired, the browser has many decisions still to make before actually sending the request:

• Which version of the HTTP protocol should be used?
• Which network protocol should be used: TCP or QUIC?
• Can an existing connection be reused?
• Can we trust the server?

All of these decisions depend on later stages of the request that will be covered in other articles in the series.

Ways That Attackers Try to Fool You

Even without sending a single request, there are ways that attackers already try to fool you into accessing their site or even capture your requests without you noticing that anything is wrong.

User Info-Based Attacks

One way attackers trick users in phishing attacks is by using the user information part of the URL to make it look like you are accessing one site, but in reality, your data is being sent to another place.

Take for example this URL: https://google.com@site.com. If you only give a fast look to it and don’t process the @ in the URL, you probably would think that you are accessing google.com, but look at the diagram below. google.com is the user information part of this URL, while the real domain being accessed is site.com.

This became such a problem that now, if you click on a URL that has the user information provided, the security tools in your browser will alert you about this and maybe block the request.

Punycode

URLs are restricted to having only a subset of the ASCII characters, but for a more inclusive internet, RFC 3492 defines a way of encoding Unicode characters in a URL, called punycode, allowing languages that use characters with accents, or even a completely different alphabet (like Chinese or Japanese), to use their native character set on URLs.

These characters are encoded with a prefix of xn-- and then a representation of the bytes of the characters encoded using the allowed URL characters, so, for example, 😀.com is encoded as xn--e28h.com.

Attackers have been exploiting this feature to fool users by replacing characters in URLs by very similar characters of the Unicode character set, leading to URLs that look the same but are completely different.

Consider this domain: аpple.com. It looks legitimate, but, if you copy it to the address bar, you’ll see something different. The a in this domain is not an ASCII character. It is a Cyrillic а. So, this domain, under the hood, is actually xn--pple-43d.com.

This type of attack is typically paired with some kind of phishing attack. To prevent this, browsers detect suspicious characters in the URL and display the raw, punycode-encoded URL, so you see the weird URL and it’s easier to detect something is wrong.

URL Parser Confusion

Since there are multiple algorithms to parse URLs used in browsers and programming languages standard libraries, the same URL can be parsed in different ways in the path of the request from the browser to the server. This leads to different systems taking actions based on different interpretations of the same URL.

Consider the URL https://good.com\@evil.com/. Different parsers interpret the \ character as having different meanings. The WHATWG specified parser treats \ the same as / for some schemes, like http and https, so everything after the \ is treated as the path part of the URL. But, implementations that follow the RFC 3986 definition on how to parse a URL, don’t treat the \ as a special character, treating it as just another character, and interpret good.com\ as the user information part of the URL and resolve the domain to be evil.com.

This can be checked with the implementations used by the standard libraries of JavaScript and Python. JavaScript uses the WHATWG algorithm and Python the RFC 3986 definition, so the result of parsing is different and the decisions made based on these implementations need to take that into account.

new URL("https://good.com\\@evil.com/").hostname;
// → 'good.com'

from urllib.parse import urlsplit
urlsplit("https://good.com\\@evil.com/").hostname
# → 'evil.com'

Proxy Attacks

Attackers take advantage of the capabilities of proxies to sit between your device and the applications you try to access and capture your data.

This can happen in many ways. One of them is hijacking the process of proxy auto-discovery to inject some kind of malicious proxy in the way. By doing that, attackers can capture the data you transfer without encryption to websites and, in some advanced attacks, even the encrypted data when the proxy intercepts the TLS certificates from the application servers and is able to replace them with a spoofed one, and the browser can be made to trust this certificate, allowing the proxy to decrypt the data in transit. This typically requires either that the attacker has already compromised the machine to install a malicious root certificate, or that the user is tricked into skipping the browser’s certificate warning. We’ll cover how certificates and data encryption work in a later article on TLS.

Still, No Request Has Been Sent Yet

As we’ve explored in this article, the browser does a lot of work even before opening the connection to the application server:

1. Understanding your intentions with what you typed
2. Parsing all the URL parts from the text string
3. Filling in the missing data
4. Checking if the request is allowed to proceed
5. Seeing whether the request can be handled locally
6. Deciding if a proxy should be used
7. Checking if the domain’s IP address is already known
8. Preparing for the later decisions in the pipeline

Throughout all these stages, security is a big concern because they have a bigger attack surface than it looks at first glance, like we saw: HSTS automatically upgrading connections to HTTPS, the trust required for service worker registration, the broad permissions extensions require, the various URL tricks attackers use to fool you, and proxies being exploited to intercept and manipulate requests and responses.

Despite all this work, the browser does not have an IP address to connect to and no byte has been exchanged over the network.

As mentioned, that’s exactly what the next article covers: translating domains into IP addresses through DNS.