psf / psf/requests

Requests does not preserve supplied CookiePolicy or CookieJar

Open
#5,217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Recently, I have run into some niche issues with cookies where servers are sending cookies that they themselves do not accept. An example is in https://github.com/psf/requests/issues/4592 or where a 'Set-Cookie' header has a value that contains ASCII control codes -- r'[\x01-\x1F\x7F]'. I fully accept that it is not within the scope of requests to validate cookies (nor should it be), which is why I have come to a solution of creating a new CookiePolicy which will refuse to set cookies that are similar to the previous 'Set-Cookie' example.

Unfortunately, in using the new policy with the default Session class it appears that the policy is not preserved while making requests. In investigating further, it looks like requests just overrides whatever CookieJar or CookiePolicy is set by the user to be RequestsCookieJar instead of preserving the supplied CookieJar.

Since this is heavily reliant on what the server sends, I tried to supply as complete and self-contained of an example as possible below (which requires a quickly spun-up server), but let me know if the issue needs further explanation.

Additionally, I would be more than happy to submit a PR that supplies a fix and adequate tests if needed (though after validation that this is not intended behavior, of course).

Expected Result

I expect for the cookie policy to be preserved while making requests within a session or requests at all. That is, there should be no AssertionErrors or printed statements in the below example.

Actual Result

Failed to preserve cookie policy with set_policy
Failed to preserve cookie policy with RequestsCookieJar
Failed to preserve cookie policy with NewCookieJar

Reproduction Steps

Spin up a local server that sends a weird cookie


from flask import Flask, redirect, make_response, request, url_for

app = Flask(__name__)

@app.route('/plain_req')
def plain_req():
    resp = make_response('Single response')
    resp.headers['Set-Cookie'] = "bad_cookie=\x01; path=/; HttpOnly"
    return resp

@app.route('/redirect_1')
def redirect_1():
    return redirect(url_for('redirect_2'), code=302)

@app.route('/redirect_2')
def redirect_2():
    resp = make_response(redirect(url_for('redirect_3'), code=302))
    resp.headers['Set-Cookie'] = "bad_cookie=\x01; path=/; HttpOnly"
    return resp

@app.route('/redirect_3')
def redirect_3():
    if '\x01' in request.cookies.get('bad_cookie'):
        resp = make_response('Client Error: Bad cookies recieved', 401)
    else:
        resp = make_response('It all worked, yay!', 200)
    return resp

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8000)

Test for the cookie (will raise AssertionError in either the context manager or

import requests
from requests.cookies import RequestsCookieJar
from http.cookiejar import DefaultCookiePolicy, CookieJar

class NewCookiePolicy(DefaultCookiePolicy):

    def set_ok(self, cookie, request):
        valid = super(NewCookiePolicy, self).set_ok(cookie, request)
        if '\x01' in cookie.value:
            return False
        return valid

def test_session(session):
    # A plain request does not work
    req = session.get('http://127.0.0.1:8000/plain_req')
    assert '\x01' not in req.cookies.items()[0][1]  # cookie still around
    assert isinstance(req.cookies._policy, NewCookiePolicy)  # Policy not preserved

    # Requests with multiple redirects do not
    req = session.get('http://127.0.0.1:8000/redirect_1')
    assert '\x01' not in req.history[1].cookies.items()[0][1]  # cookie still around
    assert isinstance(req.history[1].cookies._policy, NewCookiePolicy)  # Policy not preserved

    # Comment out the above assertion to try the context manager
    with requests.Session() as s:
        s.cookies.set_policy(NewCookiePolicy())
        context_req = s.get('http://127.0.0.1:8000/redirect_1')
        assert '\x01' not in req.history[1].cookies.items()[0][1]  # cookie still around
        assert isinstance(context_req.history[0].cookies._policy, NewCookiePolicy)  # Policy not preserved

policy_session = requests.Session()
policy_session.cookies.set_policy(NewCookiePolicy())
try:
    test_session(policy_session)
except AssertionError:
    print("Failed to preserve cookie policy with set_policy")

req_jar_session = requests.Session()
req_jar_session.cookies = RequestsCookieJar(policy=NewCookiePolicy())
try:
    test_session(req_jar_session)
except AssertionError:
    print("Failed to preserve cookie policy with RequestsCookieJar")

NewCookieJar = CookieJar(policy=NewCookiePolicy())

new_jar_session = requests.Session()
new_jar_session.cookies = NewCookieJar
try:
    test_session(new_jar_session)
except AssertionError:
    print("Failed to preserve cookie policy with NewCookieJar")

System Information

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "18.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.25.4"
  },
  "using_pyopenssl": false
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the self-contained Flask reproduction and test_session example to observe cookie handling across a plain request and redirects. Trace how Session requests and response cookies handle a supplied CookieJar or CookiePolicy; done means the policy remains in effect without the reported AssertionErrors or unwanted cookie values, with regression coverage added.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.