psf / psf/requests

Merging of default HTTP headers with specified headers breaks the defined ordering

Open
#5,814 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Even if headers are defined in an OrderedDict, their order might change if they coincide with the default_headers set in utils.py, namely User-Agent, Accept-Encoding, Accept, and Connection.

Expected Result

The order of the headers should not change when passed as an OrderedDict. So for example:

import requests
from collections import OrderedDict

headers = OrderedDict([
    ("Accept",   '1'),
    ("Accept-Encoding",   '2'),
    ("User-Agent",   '3'),
])
r = requests.get('http://www.example.com', headers=headers)

should result in the Accept header coming first, followed by Accept-Encoding and finally User-Agent.

Actual Result

The User-Agent comes first, followed by Accept-Encoding and then Accept

Reason

The default headers --- as set by requests.utils.default_headers() --- are merged with the user specified headers by the function requests.sessions.merge_settings. This function first takes the default headers and then updates that with the user specified headers. This ensures that user specified headers overwrite the default ones.

However, the update process does not change the order of the default headers. So when setting User-Agent after Accept, this will not be taken into account as both headers are already in the default headers and in reverse order.

A fix to merge_settings could be to take the user specified headers first, and then iterate over the default headers and add them in if their are misssing.

Workaround

If one wants exactly the headers specified, one can first create a Session and remove the default headers:

s = requests.Session()
s.headers = {}
r = s.get(url, headers=headers, ...)

Having a way to change the default headers on the requests module would be a welcome addition. So one could change, for example, the user agent in one place and have it affect all requests calls that follow.

Reproduction Steps

import requests
from collections import OrderedDict

headers = OrderedDict([
    ("Accept",   '1'),
    ("Accept-Encoding",   '2'),
    ("User-Agent",   '3'),
])
r = requests.get('http://www.example.com', headers=headers)

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "4.0.0"
  },
  "cryptography": {
    "version": "3.3.1"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.5"
  },
  "platform": {
    "release": "5.8.0-50-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010109f",
    "version": "20.0.1"
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.26.4"
  },
  "using_pyopenssl": true
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in requests.sessions.merge_settings and compare its header merge behavior with requests.utils.default_headers(). Reproduce the OrderedDict example and verify that user-specified header order is preserved while missing default headers remain available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.