Requests does not preserve supplied CookiePolicy or CookieJar
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Recently, I have run into some niche issues with cookies where servers are sending cookies that they themselves do not accept. An example is in https://github.com/psf/requests/issues/4592 or where a 'Set-Cookie' header has a value that contains ASCII control codes -- r'[\x01-\x1F\x7F]'. I fully accept that it is not within the scope of requests to validate cookies (nor should it be), which is why I have come to a solution of creating a new CookiePolicy which will refuse to set cookies that are similar to the previous 'Set-Cookie' example.
Unfortunately, in using the new policy with the default Session class it appears that the policy is not preserved while making requests. In investigating further, it looks like requests just overrides whatever CookieJar or CookiePolicy is set by the user to be RequestsCookieJar instead of preserving the supplied CookieJar.
Since this is heavily reliant on what the server sends, I tried to supply as complete and self-contained of an example as possible below (which requires a quickly spun-up server), but let me know if the issue needs further explanation.
Additionally, I would be more than happy to submit a PR that supplies a fix and adequate tests if needed (though after validation that this is not intended behavior, of course).
Expected Result
I expect for the cookie policy to be preserved while making requests within a session or requests at all. That is, there should be no AssertionErrors or printed statements in the below example.
Actual Result
Failed to preserve cookie policy with set_policy
Failed to preserve cookie policy with RequestsCookieJar
Failed to preserve cookie policy with NewCookieJar
Reproduction Steps
Spin up a local server that sends a weird cookie
from flask import Flask, redirect, make_response, request, url_for
app = Flask(__name__)
@app.route('/plain_req')
def plain_req():
resp = make_response('Single response')
resp.headers['Set-Cookie'] = "bad_cookie=\x01; path=/; HttpOnly"
return resp
@app.route('/redirect_1')
def redirect_1():
return redirect(url_for('redirect_2'), code=302)
@app.route('/redirect_2')
def redirect_2():
resp = make_response(redirect(url_for('redirect_3'), code=302))
resp.headers['Set-Cookie'] = "bad_cookie=\x01; path=/; HttpOnly"
return resp
@app.route('/redirect_3')
def redirect_3():
if '\x01' in request.cookies.get('bad_cookie'):
resp = make_response('Client Error: Bad cookies recieved', 401)
else:
resp = make_response('It all worked, yay!', 200)
return resp
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8000)
Test for the cookie (will raise AssertionError in either the context manager or
import requests
from requests.cookies import RequestsCookieJar
from http.cookiejar import DefaultCookiePolicy, CookieJar
class NewCookiePolicy(DefaultCookiePolicy):
def set_ok(self, cookie, request):
valid = super(NewCookiePolicy, self).set_ok(cookie, request)
if '\x01' in cookie.value:
return False
return valid
def test_session(session):
# A plain request does not work
req = session.get('http://127.0.0.1:8000/plain_req')
assert '\x01' not in req.cookies.items()[0][1] # cookie still around
assert isinstance(req.cookies._policy, NewCookiePolicy) # Policy not preserved
# Requests with multiple redirects do not
req = session.get('http://127.0.0.1:8000/redirect_1')
assert '\x01' not in req.history[1].cookies.items()[0][1] # cookie still around
assert isinstance(req.history[1].cookies._policy, NewCookiePolicy) # Policy not preserved
# Comment out the above assertion to try the context manager
with requests.Session() as s:
s.cookies.set_policy(NewCookiePolicy())
context_req = s.get('http://127.0.0.1:8000/redirect_1')
assert '\x01' not in req.history[1].cookies.items()[0][1] # cookie still around
assert isinstance(context_req.history[0].cookies._policy, NewCookiePolicy) # Policy not preserved
policy_session = requests.Session()
policy_session.cookies.set_policy(NewCookiePolicy())
try:
test_session(policy_session)
except AssertionError:
print("Failed to preserve cookie policy with set_policy")
req_jar_session = requests.Session()
req_jar_session.cookies = RequestsCookieJar(policy=NewCookiePolicy())
try:
test_session(req_jar_session)
except AssertionError:
print("Failed to preserve cookie policy with RequestsCookieJar")
NewCookieJar = CookieJar(policy=NewCookiePolicy())
new_jar_session = requests.Session()
new_jar_session.cookies = NewCookieJar
try:
test_session(new_jar_session)
except AssertionError:
print("Failed to preserve cookie policy with NewCookieJar")
System Information
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.8"
},
"implementation": {
"name": "CPython",
"version": "3.7.4"
},
"platform": {
"release": "18.7.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.22.0"
},
"system_ssl": {
"version": "1000213f"
},
"urllib3": {
"version": "1.25.4"
},
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the self-contained Flask reproduction and test_session example to observe cookie handling across a plain request and redirects. Trace how Session requests and response cookies handle a supplied CookieJar or CookiePolicy; done means the policy remains in effect without the reported AssertionErrors or unwanted cookie values, with regression coverage added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100