Still cannot prevent requests library from encoding
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
I have the same problem as https://github.com/psf/requests/issues/5964, but the solution does not work for me.
I need to send unescaped URL requestes to a server. The purpose is to evoke response code 400 to test the firewall configuration.
Expected Result
The following code should (according to https://github.com/psf/requests/issues/5964) send the unescaped URL:
import requests
s = requests.Session()
# Using Burp Suite Proxy to examine request being sent.
s.proxies = {
"http" : "http://127.0.0.1:8080",
"https" : "http://127.0.0.1:8080",
}
# Do not verify certificate.
s.verify = False
# Solution from https://github.com/psf/requests/issues/5964
base_url = 'https://www.example.com/search'
query = '?date_range=2017-01-01|2017-03-01'
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)
Actual Result
The print(resp.request.url) prints https://www.example.com/search?date_range=2017-01-01|2017-03-01 as expected.
But what is really transmitted is the escaped URL, https://www.example.com/search?date_range=2017-01-01%7C2017-03-01 as seen in this Burp Suite screenshot:

Reproduction Steps
Use the python code above and a way of choice to capture the actual response being sent (e.g. Burp Suite proxy).
You can also use curl to send an actually unescaped response:
curl "https://www.example.com/search?date_range=2017-01-01|2017-03-01" -x 127.0.0.1:8080 --insecure

System Information
$ python -m requests.help
{
{
"chardet": {
"version": "3.0.4"
},
"charset_normalizer": {
"version": "2.0.12"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.8"
},
"implementation": {
"name": "CPython",
"version": "3.8.10"
},
"platform": {
"release": "5.14.0-1034-oem",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.27.1"
},
"system_ssl": {
"version": "1010106f"
},
"urllib3": {
"version": "1.25.8"
},
"using_charset_normalizer": false,
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the provided Python reproduction through Burp Suite and compare the prepared URL with the bytes transmitted on the wire, then inspect the request path through Requests and its urllib3 dependency. Done means determining whether the literal pipe can be transmitted without encoding while preserving normal request behavior, with a regression test covering the observed case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100