Request for clarification: URL parsing behavior in Requests (RFC 3986 vs WHATWG)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Hello Requests maintainers,
Thank you for maintaining Requests and for the extensive discussions around URL parsing behavior over the years.
I am seeking clarification regarding which specification(s) Requests follows for URL parsing, particularly for HTTP(S) URLs, as I have observed behaviors that appear to align with different standards.
In previous discussions, Requests developers have noted that the library primarily follows RFC-based specifications. For example, in https://github.com/psf/requests/issues/5886, it is stated that:
RFC 3986 is the standard that Requests and urllib3 primarily use as a reference for their URL parsers.
This suggests that Requests' URL parsing logic is grounded in RFC 3986, with HTTP semantics further informed by RFC 7230.
However, I have observed a behavior in Requests that seems closer to the WHATWG URL Standard.
Specifically, when parsing HTTP URLs whose hostname contains a backslash (), Requests appears to treat \ as a delimiter equivalent to /, effectively separating the authority from the path.
This behavior is consistent with the WHATWG URL Standard's definition of the authority parsing state (see: https://url.spec.whatwg.org/#authority-state).
By contrast: RFC 3986 treats \ as a disallowed character in URLs and does not define it as a structural delimiter. Many RFC-oriented parsers (e.g., urllib.parse.urlparse) either reject such URLs or treat \ as an ordinary character, rather than as a path/authority separator.
Question
Could you please clarify:
Whether Requests intentionally references or aligns with the WHATWG URL Standard for any part of its URL parsing behavior (especially hostname handling)?
Or whether the observed behavior is still considered RFC 3986–based, but with pragmatic extensions or legacy compatibility considerations?
Understanding this would be very helpful for accurately characterizing Requests' parsing semantics and for reasoning about cross-component URL parsing consistency.
Thank you very much for your time and clarification.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the cited issue #5886 and comparing the observed behavior with the RFC 3986, RFC 7230, and WHATWG URL references linked here. No source file or test is identified; the issue is done when the maintainers clarify which specification or compatibility behavior Requests intentionally follows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100