psf / psf/requests

replace `get_encoding_from_headers` default with `None`

Open
#5,084 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.5k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

This issue a suggestion for handling partial elements of issues #1737 and #2086 (#1774 and others too). I'm bringing this up in a separate ticket because it is a specific fix for multiple tickets and I am willing to write a PR for it in requests3 and requests2. (i would have just gone and generated a PR, but I can't figure out what the actual master for requests3 actually is)

my proposal is to do the following (some of which have already been discussed as possibilities):

alter get_encoding_from_headers()

replace the default value of get_encoding_from_headers with None when "text" is in the content_type header. currently this is "ISO-8859-1"

https://github.com/kennethreitz/requests/blob/c501ec986daa4961cd9dee370b5d45ff2e524b37/requests/utils.py#L492-L493

add Response.detected_encoding attribute

extend build_response and Response to additionally stash the header decoding in response.detected_encoding. this will allow the detected value to remain on the response object if response.encoding is changed by the consumer.

add Response.text_encoding property

add a text_encoding property to Response , which offloads some of the logic now in text, and will return "ISO-8859-1"

https://github.com/kennethreitz/requests/blob/c501ec986daa4961cd9dee370b5d45ff2e524b37/requests/models.py#L835-L858

alter Response.text to use the new .text_encoding attribute

update the text property to use the new .text_encoding attribute. this just ties together all the above.

Goals

The goal of these changes is to positively identify situations where no response encoding was declared by the server. The backwards incompatibility is that None will now be returned when inspecting response.encoding when there is no encoding for the response - however the effective encoding used to generate .text will be available in response.text_encoding .

This still maintains compliance with RFC 2616, because .text will still interpret the lack of a declared encoding as having the required default charset of ISO-8859-1. This simply allows developers to know when ISO-8859-1 is being used in a default or explicit manner.

This approach doesn't address all aspects of those issues or close those tickets, but it offers a solution to many of the needs and concerns with some minor changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with requests/utils.py at get_encoding_from_headers and requests/models.py around Response.text, then trace build_response and the Response attributes involved. Compare the proposal with issues #1737, #2086, and #1774. Done means distinguishing an absent declared encoding from an explicit one while preserving ISO-8859-1 as the effective default for text decoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.