psf / psf/requests

Still cannot prevent requests library from encoding

Open
#6,115 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

I have the same problem as https://github.com/psf/requests/issues/5964, but the solution does not work for me.
I need to send unescaped URL requestes to a server. The purpose is to evoke response code 400 to test the firewall configuration.

Expected Result

The following code should (according to https://github.com/psf/requests/issues/5964) send the unescaped URL:

import requests
s = requests.Session()

# Using Burp Suite Proxy to examine request being sent.
s.proxies = { 
    "http"  : "http://127.0.0.1:8080",
    "https" : "http://127.0.0.1:8080",
}
# Do not verify certificate.
s.verify = False

# Solution from https://github.com/psf/requests/issues/5964
base_url = 'https://www.example.com/search'
query = '?date_range=2017-01-01|2017-03-01'
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)

Actual Result

The print(resp.request.url) prints https://www.example.com/search?date_range=2017-01-01|2017-03-01 as expected.
But what is really transmitted is the escaped URL, https://www.example.com/search?date_range=2017-01-01%7C2017-03-01 as seen in this Burp Suite screenshot:
image

Reproduction Steps

Use the python code above and a way of choice to capture the actual response being sent (e.g. Burp Suite proxy).
You can also use curl to send an actually unescaped response:

curl "https://www.example.com/search?date_range=2017-01-01|2017-03-01" -x 127.0.0.1:8080 --insecure

image

System Information

$ python -m requests.help
{
  {
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.10"
  },
  "platform": {
    "release": "5.14.0-1034-oem",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.25.8"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the provided Python reproduction through Burp Suite and compare the prepared URL with the bytes transmitted on the wire, then inspect the request path through Requests and its urllib3 dependency. Done means determining whether the literal pipe can be transmitted without encoding while preserving normal request behavior, with a regression test covering the observed case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.