Referrer and cache control APIs for fetch()

Around a year ago, we wrote about the new fetch() API.  The WHATWG Fetch API provides a modern way to fetch network resources and gives you fine grained control over the details of the request and response.  If you’re not familiar with the Fetch API, it would be a nice idea to read about it before proceeding.

We have recently implemented a few new additions to the Fetch API, and in this post I will give an overview of them and include examples of how they can help you develop your web applications.

Referrer control APIs

Using fetch(), you can now control the HTTP request referrer and referrer policy. The HTTP Referer [sic] header is a (misspelled!) header that allows a target page to know what source page the user is coming from (for example, by clicking a link on that page). This is useful for example for gathering analytics data about where your web site users are coming from.

The referrer policy is a new W3C specification which we have been implementing in Firefox that allows the page to provide the browser with a policy that lets the page have more control over how the Referer header is set.  There are a few different policy states, each with a specific goal in mind.  Here is a summary.

  • “no-referrer” prevents sending any Referer header.  This can be useful when you want to hide the Referer header for privacy reasons.  For example, some search engines add information about the user’s search phrase among other things to the URL, and they may not want to leak the user’s search phrase to the search result web sites that the user clicks on.  The “no-referrer” referrer policy could be used for that purpose.
  • “no-referrer-when-downgrade” is similar to “no-referrer” with the exception that the Referer header is only omitted when navigating from a secure context to a non-secure one.  For instance, with the search engine example above, if your privacy concern is limited to people monitoring the HTTP traffic instead of the target website, you can use the “no-referrer-when-downgrade” policy. In this case, if a search result links to a secure context, the browser would send the Referer header but if the target web site is an insecure HTTP site, the browser will refuse the send the Referer header in clear text.  This is the default policy if an explicit policy has not been specified.
  • “origin” will make the browser only include the referring origin and not the full URL in the Referer header.  For example, if you want the target web sites to be able to tell that the user is coming from your search results page without revealing the full URL, you can use “origin”. In this case, the browser would strip away anything after the domain name in the URL sent in the Referer header.
  • “origin-when-cross-origin” is similar to “origin” except that it will only strip out the full URL when navigating across origins.  For example, suppose you want to only include the origin for your search result pages (which we assume to be cross origin if your site is doing a normal web search), but send the full referrer to your own internal pages. This lets your own analytics software know how your users navigate across the pages of your site. In this case, “origin-when-cross-origin” is the right policy to choose.
  • “unsafe-url” causes the browser to send the full URL (sans any associated user name, password or fragment) to all pages that the user navigates to, no matter whether they’re cross origin and/or secure.  The real reason why this is called unsafe is that this will reveal the full URL to any target web page, which raises privacy concerns such as those the examples above try to address.  You should consider using a different referrer policy if possible.

Right now, in Firefox you can use an <meta name=referrer> element on your page to set a global referrer policy for all network requests initiated from the page.  We are also working on implementing the per-element referrer policy attributes that can be useful when you want to use a different referrer policy for a specific element (such as an <img>).  With the new APIs introduced here, you can also control the referrer and the referrer policy for the resources downloaded using fetch().

The following code examples show a few examples of how you can use these new fetch() features.

  // Let’s assume that the code below runs on https://example.site/page.html

  // Download a json but don't reveal who is downloading it
  fetch("sneaky.json", {referrerPolicy: "no-referrer"})
    .then(function(response) { /* consume the response */ });

  // Download a json but pretend another page is downloading it
  fetch("sneaky.json", {referrer: "https://example.site/fake.html"})
    .then(function(response) { /* consume the response */ });

  // You can only set same-origin referrers.
  fetch("sneaky.json", {referrer: "https://cross.origin/page.html"})
    .catch(function(exc) {
      // exc.name == "TypeError"
      // exc.message == "Referrer URL https://cross.origin/page.html cannot be cross-origin to the entry settings object (https://example.site)."
    });

  // Download a potentially cross-origin json and don't reveal
  // the full referrer URL across origins
  fetch(jsonURL, {referrerPolicy: "origin-when-cross-origin"})
    .then(function(response) { /* consume the response */ });

  // Download a potentially cross-origin json and reveal a
  // fake referrer URL on your own origin only.
  fetch(jsonURL, {referrer: "https://example.site/fake.html",
                  referrerPolicy: "origin-when-cross-origin"})
    .then(function(response) { /* consume the response */ });

  // Override sending the document global referrer policy set using
  // to send the full referrer URL.
  // Be careful!
  fetch(jsonURL, {referrerPolicy: "unsafe-url"})
    .then(function(response) { /* consume the response */ });

If your site uses service workers, then you can examine the referrer and the referrer policy that accompanies a fetched resource. Look inside the fetch event handler using the referrer and referrerPolicy attributes of Request objects.

This API will be available in Firefox 47, which is currently available in the developer edition release channel for testing.

Fetch cache control APIs

The resources downloaded through fetch(), similar to other resources that the browser downloads, are subject to the HTTP cache.  This is usually fine, since it means that if your browser has a cached copy of the response to the HTTP request. It can use the cached copy instead of wasting time and bandwidth re-downloading from a remote server.

However, there are cases where you would want some control over whether the browser’s HTTP cache is used. You can ensure that you’re getting a fresh response no matter what’s in the browser’s HTTP cache by cache busting the URL of the resources you want to to fetch into the service worker controlled cache. This is typically done by appending a parameter such as 'cache-bust=' + Date.now() to the URL before downloading it, which is quite ugly.  There is now a better way to do this, using the fetch cache control API.

The idea behind this API is specifying a caching policy for fetch to explicitly indicate how and when the browser HTTP cache should be consulted.  It’s important to have a good understanding of the HTTP caching semantics in order to use these most effectively.  There are many good articles on the web such as this one that describe these semantics in detail.  There are currently five different policies that you can choose from.

  • “default” means use the default behavior of browsers when downloading resources.  The browser first looks inside the HTTP cache to see if there is a matching request.  If there is, and it is fresh, it will be returned from fetch().  If it exists but is stale, a conditional request is made to the remote server and if the server indicates that the response has not changed, it will be read from the HTTP cache.  Otherwise it will be downloaded from the network, and the HTTP cache will be updated with the new response.
  • “no-store” means bypass the HTTP cache completely.  This will make the browser not look into the HTTP cache on the way to the network, and never store the resulting response in the HTTP cache.  Using this cache mode, fetch() will behave as if no HTTP cache exists.
  • “reload” means bypass the HTTP cache on the way to the network, but update it with the newly downloaded response.  This will cause the browser to never look inside the HTTP cache on the way to the network, but update the HTTP cache with the downloaded response. Future requests can use that updated response if appropriate.
  • “no-cache” means always validate a response that is in the HTTP cache even if the browser thinks that it’s fresh.  This will cause the browser to look for a matching request in the HTTP cache on the way to the network.  If such a request is found, the browser always creates a conditional request to validate it even if it thinks that the response should be fresh.  If a matching cached entry is not found, a normal request will be made.  After a response has been downloaded, the HTTP cache will always be updated with that response.
  • “force-cache” means that the browser will always use a cached response if a matching entry is found in the cache, ignoring the validity of the response.  Thus even if a really old version of the response is found in the cache, it will always be used without validation.  If a matching entry is not found in the cache, the browser will make a normal request, and will update the HTTP cache with the downloaded response.

Let’s look at a few examples of how you can use these cache modes.

  // Download a resource with cache busting, to bypass the cache
  // completely.
  fetch("some.json", {cache: "no-store"})
    .then(function(response) { /* consume the response */ });

  // Download a resource with cache busting, but update the HTTP
  // cache with the downloaded resource.
  fetch("some.json", {cache: "reload"})
    .then(function(response) { /* consume the response */ });

  // Download a resource with cache busting when dealing with a
  // properly configured server that will send the correct ETag
  // and Date headers and properly handle If-Modified-Since and
  // If-None-Match request headers, therefore we can rely on the
  // validation to guarantee a fresh response.
  fetch("some.json", {cache: "no-cache"})
    .then(function(response) { /* consume the response */ });

  // Download a resource with economics in mind!  Prefer a cached
  // albeit stale response to conserve as much bandwidth as possible.
  fetch("some.json", {cache: "force-cache"})
    .then(function(response) { /* consume the response */ });

This API is planned for release in Firefox 48, and is currently available in Firefox Nightly for testing.

About Ehsan Akhgari

Ehsan has worked on various parts of Firefox since 2006. These days he spends most of his time working on Gecko and Web APIs.

More articles by Ehsan Akhgari…


14 comments

  1. Chen Zhixiang

    I’ve recently implemented a feature based on chromium, which forced to cache the http response data, when parser sees a in html early. (And all the sub resources will be forced to stored in diskcache)

    So my question is, why no “force-store”, since there is a “force-cache”?

    March 23rd, 2016 at 03:09

    1. Ehsan Akhgari

      What would “force-store” do that you cannot achieve with the existing cache modes?

      March 23rd, 2016 at 08:42

      1. Chen Zhixiang

        I mean, “force-store” is the opposite of “no-store”, to bypass the default http cache policy to cache the response data in the local diskcache.

        With “force-store” and “force-cache”, we can implement a simple “offline web apps” mode, where everything is controlled by use himself.

        March 23rd, 2016 at 20:06

        1. Ehsan Akhgari

          I think the cache mode that you’re asking for is “reload”. It will bypass the HTTP cache on the way to the network, and will store the newly fetched data in the HTTP cache for future usage. Does that make sense?

          March 24th, 2016 at 08:38

          1. Chen Zhixiang

            No, “reload” is not the same meaning as “force-store” which i’m suggesting here.

            “reload“ means reload from network,but if http server response Cache-Control header says ”no-store“, “reload” cannot do diskcache according to http cache policy.

            While “force-store”, which i’m suggesting here, will bypass the default http cache policy, force to store the response data(headers and body) into diskcache, even server side may say “Cache-Control: no-store”

            March 24th, 2016 at 19:01

          2. Ehsan Akhgari

            OK, I see the distinction now. If you have a use case for this, please file an issue here: https://github.com/whatwg/fetch/issues.

            March 28th, 2016 at 07:44

  2. Eric Lawrence

    Cool, thanks for writing!

    duplicate sentence:

    “However, there are cases where you would want some control over whether the browser’s HTTP cache is used”

    March 23rd, 2016 at 22:02

    1. Ehsan Akhgari

      Thanks, fixed!

      March 24th, 2016 at 08:36

  3. Koemsie Ly

    Thanks for you information!

    March 24th, 2016 at 07:06

  4. Gerben

    Wouldn’t it make sense to have the “reload” option also do the If-Modified-Since, like “no-cache” does?

    March 26th, 2016 at 12:47

    1. Ehsan Akhgari

      I don’t think so. Sending the If-Modified-Since header is part of the HTTP cache validation which is a process designed to determine if an HTTP cached response is fresh enough to be used to respond to a request with, however “reload” acts as if there is no HTTP cache at all on the way to the network, so there would be no response to validate.

      I think the behavior that you want is better supported by using “default” or “no-cache”, the former validates a response (if a matching one is found) if needed, and the latter validates a response unconditionally before using it.

      March 28th, 2016 at 07:48

      1. Gerben

        “fresh enough” is not really the right term I think. If-modified-since check if the local copy is still the latest version.
        “Fresh enough” would be more appropriate to explain the “Expires” header.

        The “no-cache” method would be better, in that it check if the local copy is still the latest, but, from I infer, doesn’t store it in the local cache.

        If so, there is no way to have fetch ignore the “Expires” header, but still do the “If-modified-since” and write any changes to the local cache.

        E.g. a webcam website. You want to check if there is a new image, but you don’t want to waste bandwidth downloading the same image twice. (Though in this case, the server has to also send an “expires” header too long into the future)

        March 29th, 2016 at 08:04

        1. Ehsan Akhgari

          Here is the default behavior (I’m ignoring ETag and the respective If-None-Match checks that are the other half of HTTP cache validation): If the cached resource has an Expires header that is in the future, it is considered fresh enough. If the Expires header date is in the past, an If-Modified-Since request is generated to validate that the cached entry is “fresh enough”. This is the “default” behavior.

          “no-cache” changes this so that irrespective of what the Expires header indicates, the browser *always* sends an If-Modified-Since request.

          With a server that is configured to send the right Expires header, the “default” behavior is always sufficient to ensure that you get a fresh response either from the cache or from the network in the case where the server indicates the response has been modified in response to the If-Modified-Since request. However, if the server is known to set unrealistic Expires headers (for example a webcam image server that sends an Expires header too far into the future as you indicated) you want to use “no-cache” to ensure that the server will get a chance to send a new response even if a previous response’s Expires header would make the browser believe its cached response is fresh enough otherwise.

          With regard to your point about “no-cache” not updating the HTTP cache with the downloaded response, that’s not correct, as the explanation in the article indicates. If a response is downloaded in response to a “no-cache” request fetch, it is always stored in the HTTP cache.

          Hope this helps clarify things!

          March 29th, 2016 at 08:39

          1. Gerben

            Now it’s clear to me. Thank you for taking the time to explain.

            March 29th, 2016 at 08:51

Comments are closed for this article.