psf / psf/requests

requests can't properly handle redirects if the response body is encoded in something else than 'utf8'

Open
#4,926 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Just like in the topic. The response body is encoded in iso-8859-2 and the location happens to contain non-ascii character so that it results in UnicodeDecodeError being thrown.

Expected Result

Flawless execution of the code.

Actual Result

UnicodeDecodeError

Reproduction Steps

import requests
requests.get("http://www.biblia.deon.pl/ksiega.php?id=3")

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": "2.3"
  }, 
  "idna": {
    "version": "2.7"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.15+"
  }, 
  "platform": {
    "release": "4.18.0-13-generic", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "1010100f", 
    "version": "18.0.0"
  }, 
  "requests": {
    "version": "2.19.0"
  }, 
  "system_ssl": {
    "version": "1010100f"
  }, 
  "urllib3": {
    "version": "1.23"
  }, 
  "using_pyopenssl": true
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided requests.get reproduction against the biblia.deon.pl URL and trace the redirect handling path. Check how the response encoding and non-ASCII Location value interact, then add coverage for an iso-8859-2 response with a redirect. Done means the request follows the redirect without raising UnicodeDecodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.