Proposal: Easier request/response rewrites

Hi all,

For those that don't know, I'm the tech lead for Cloudflare Workers, which implements the Service Workers API (but runs code on Cloudflare's servers rather than in the browser).

In general the Service Workers and Fetch APIs have been very good for us (certainly better than what we would have ended up with had we invented our own). But, based on user feedback we are finding some pain points. This is the first of a few proposals I'll be making to solve these issues.

/cc @harrishancock who does most of the API implementation work on our team.

Problem statement

Today, making minor "rewrites" to a Request or Response (e.g. starting from an existing object and modifying just one property or header) is not very ergonomic. I'm not sure if this is common in browser Service Workers, but it is probably the dominant use case for Cloudflare Workers, so we're very interested in making it better.

For example, consider the case where I want to change the hostname, change the redirect mode to "follow" (to resolve redirects server-side), and add a header. This might look like:

addEventListener("fetch", event => {
  let request = event.request

  // Change URL.
  let url = new URL(event.request.url)
  url.hostname = "example.com"
  request = new Request(url, request)
      
  // Change the redirect mode.
  request = new Request(request, { redirect: "follow" })
      
  // Add a header.
  request.headers.add("X-Example", "foo")

  event.respondWith(fetch(request))
})

Notice how the best way to change each property is wildly different!

• The only way (AFAIK) to modify the URL while keeping everything else the same is request = new Request(newUrl, request). The fact that this works is somewhat of a fluke: we're actually duck-typing the old request object as a RequestInit, which happens to work because all the member names line up. It's unclear to me if the spec actually intended for this to work, but without this trick, there's no way to construct the new request without enumerating every request property individually, which is error-prone and not future-proof.
• We modify redirect in the intended way, by passing the old request as the first parameter to the new request's constructor, and passing a RequestInit containing only redirect. This is fine.
• Modifying headers via RequestInit is inconvenient because it replaces all of the headers. Since we don't want to remove the existing headers, we'd need to make a copy Headers object first, modify the copy, then pass that in RequestInit. It turns out, though, that once we've made our own Request object, we can just modify its headers directly. This is convenient, but weirdly inconsistent: properties like url and redirect cannot be modified post-construction.

When it comes to the Response type, we have a bigger problem: you cannot pass an existing response object as the first parameter to Response's constructor, the way you can do with requests. (If you try to do so, the response object will be stringified as [object Response] and that will become the new response's body.) So, if you want to modify the status code of a Response:

addEventListener("fetch", event => {
  if (new URL(event.request.url).pathname.startsWith("/hidden/")) {
    // Mask our hidden directory by faking 404.
    event.respondWith(fetch("/404-page.html")
        .then(response => {
      // 404-page.html returns with status 200. Give it status 404.
      return new Response(response.body, {
        status: 404,
        statusText: "Not Found",
        headers: response.headers
      })
    }))
  }
})

This is bad, because if Response and ResponseInit are ever extended with a new field, that field will be inadvertently dropped during the rewrite. (We commonly see people doing Request rewrites this way, too, where it's an even bigger issue as RequestInit has quite a few fields that tend to be forgotten.)

Proposal

Let's make Request's constructor be the One True Way to rewrite requests. To that end:

• Define RequestInit.url as an alternative way to specify the URL. This field would only be used when the constructor's first parameter is an existing request object, in which case RequestInit.url overwrites the URL. (It's important that RequestInit.url is ignored when the first parameter is itself a string URL. Otherwise, existing code which rewrites URLs using the request = new Request(url, request) idiom would break.)
• Define RequestInit.setHeaders and RequestInit.appendHeaders as type record<ByteString, ByteString>. If specified, this is equivalent to calling request.headers.set() or request.headers.append() with each key/value pair after the request object is constructed.

Similarly, let's fix Response to use the same rewrite idiom:

• Allow Response's constructor to take another Response object as the first parameter, in the same way Request's constructor does today.
• Define ResponseInit.body as an alternative way to override the body, in the specific case where the constructor's first parameter is an existing Response object.
• Define ResponseInit.setHeaders and ResponseInit.appendHeaders to work the same as with requests.

Start by reading the Request and Response constructors alongside RequestInit and ResponseInit, focusing on the rewrite behavior and the proposed URL, body, setHeaders, and appendHeaders fields. Done means the Fetch Standard defines consistent request and response rewrite semantics without dropping existing properties.