PreparedRequests can't bypass URL normalization when proxies are used
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Related to #5289, where akmalhisyam found a way to bypass URL normalization using PreparedRequests, however, the solution doesn't work when you have proxies provided.
Expected Result
This should be able to explicitly set the request URL without getting normalized (from /../something.txt to /something.txt)
url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, proxies={"http": "http://127.0.0.1"}, verify=False)
Actual Result
The code above doesn't work, this one works though:
url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, verify=False)
Reproduction Steps
Use the code in Expected Result and check your proxy request log, you will see it doesn't work
System Information
$ python -m requests.help
{
"chardet": {
"version": "5.2.0"
},
"charset_normalizer": {
"version": "2.0.12"
},
"cryptography": {
"version": "38.0.4"
},
"idna": {
"version": "3.4"
},
"implementation": {
"name": "CPython",
"version": "3.11.4"
},
"platform": {
"release": "4.4.0-19041-Microsoft",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "30000080",
"version": "21.0.0"
},
"requests": {
"version": "2.32.3"
},
"system_ssl": {
"version": "30000030"
},
"urllib3": {
"version": "2.0.4"
},
"using_charset_normalizer": false,
"using_pyopenssl": true
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at requests.Session.send and the PreparedRequest path used when proxies are supplied; compare it with the direct send path in the reproduction. Trace where prep.url is transformed before the proxy request, then add a regression test covering /../something.txt through a proxy and confirm the proxy sees the unnormalized URL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100