psf / psf/requests

PreparedRequests can't bypass URL normalization when proxies are used

Open
#6,830 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Related to #5289, where akmalhisyam found a way to bypass URL normalization using PreparedRequests, however, the solution doesn't work when you have proxies provided.

Expected Result

This should be able to explicitly set the request URL without getting normalized (from /../something.txt to /something.txt)

url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, proxies={"http": "http://127.0.0.1"}, verify=False)

Actual Result

The code above doesn't work, this one works though:

url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, verify=False)

Reproduction Steps

Use the code in Expected Result and check your proxy request log, you will see it doesn't work

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "5.2.0"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": "38.0.4"
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.4"
  },
  "platform": {
    "release": "4.4.0-19041-Microsoft",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30000080",
    "version": "21.0.0"
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30000030"
  },
  "urllib3": {
    "version": "2.0.4"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at requests.Session.send and the PreparedRequest path used when proxies are supplied; compare it with the direct send path in the reproduction. Trace where prep.url is transformed before the proxy request, then add a regression test covering /../something.txt through a proxy and confirm the proxy sees the unnormalized URL.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.