whatwg / whatwg/fetch

Handling of invalid Location header characters doesn't match browsers

Open
#883 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: http topic: redirects
Dominant language
HTML
Stars
2.3k
Forks
403
Avg merge
21h 16m
Merged PRs (30d)
4

Description

We recently regressed some handling of invalid Location headers in Chrome. In digging into that, I noticed the Fetch spec doesn't really match browsers here. I should also note I'm not very familiar with the structure of these specifications, so it's certainly possible I'm reading some of them wrong.

Fetch appears to define handling of the Location header as (https://fetch.spec.whatwg.org/#http-fetch, steps 5.2 and 5.3):

Let location be the result of extracting header list values given Location and actualResponse’s header list.
If location is a value, then set location to the result of parsing location with actualResponse’s URL.

Note, in particular, that "extracting header list values" is defined in terms of the headers ABNF. I don't see a Location ABNF in Fetch, so I assume this is intended to be the one in RFC 7231. That cites various URI-related ABNFs in RFC 3986, which implies only a subset of ASCII as allowed, and instead other things should have already been percent-encoded. Inputs that don't match appear to be an error ("extract header values" returns an error when the ABNF does not match). Is that the intended reading?

This doesn't appear to match what is actually implemented by browsers or necessary for web compatibility. In particular, we had a report that users editing ru.wikipedia.org, until recently, would be served redirects where Cyrillic characters in the path were escaped, but those in the fragment were not.

Some interesting cases to test:
Note: when testing Chrome, test with Chrome canary, specifically 75.0.3735.0 or later. 73 has a bug. 74 should have the fix in not too long but currently does not.
\xff refers to a byte with value 0xff. %ff refers to three bytes with values %, f, and f.

  • Location: https://example.com/#\xe2\x98\x83 (some unescaped UTF-8 character)
  • Location: https://example.com/#\xe2\x98\x83%e2%98%83 (mix of escaped and unescaped)
  • Location: https://example.com/#%\xe2\x98\x83 (something after a %)
  • Location: https://example.com/#\xff (UTF-8 encoding error)

Chrome (pre-73 and 75+), Firefox, and Safari all appear to resolve these to:

  • https://example.com/#%E2%98%83
  • https://example.com/#%E2%98%83%e2%98%83
  • https://example.com/#%%E2%98%83
  • https://example.com/#%FF

(I haven't tested Edge and IE.)

That last case is particularly interesting. Chrome internally percent-encodes all non-ASCII header bytes before passing the string on to URL handling logic. (This logic is very very old, so I don't know what originally motivated it.)

This seems a goofy way to specify it, but the obvious alternative doesn't seem to work. Suppose Fetch just said all Location headers were valid and you just pass the value to the URL parsing algorithm, I don't think that's what would come out. The URL parsing algorithm takes a string input, not bytes, and seems to primarily percent-encode as UTF-8. Interestingly, in the quoted text above, Fetch passes a value to that function, which is a byte sequence, not a string. And, indeed if I remove the funny preprocessing step in Chrome, %FF turns into %EF%BF%BD (U+FFFD).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Fetch Standard's HTTP-fetch steps 5.2 and 5.3, then read the linked header-value extraction and URL parsing definitions. Compare those rules with the four browser cases listed in the issue and establish the intended handling for invalid Location header bytes. Done means the specification's behavior and browser-compatible result are unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
html
Domain
documentation
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.