psf / psf/requests

Issue with Latin Characters in URL

Open
#5,258 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Getting UTF-8 when attempting to get a URL with Latin characters in the URL.

Expected Result

<Response [200]>

Actual Result

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 668, in
history = [resp for resp in gen] if allow_redirects else []
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 247, in resolve_redirects
**adapter_kwargs
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 681, in send
r._next = next(self.resolve_redirects(r, request, yield_requests=True, **kwargs))
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 149, in resolve_redirects
url = self.get_redirect_target(resp)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 115, in get_redirect_target
return to_native_string(location, 'utf8')
File "/usr/local/lib/python3.7/site-packages/requests/_internal_utils.py", line 25, in to_native_string
out = string.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 64: invalid continuation byte

Reproduction Steps

import requests
requests.get("http://www.janes.com/article/92450/south-korea-launches-philippine-navy-s-second-josé-rizal-class-frigate?from_rss=1")

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "19.0.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000212f"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": false
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the request in the issue first, then inspect the redirect flow in requests/sessions.py and decoding in requests/_internal_utils.py, as shown in the traceback. Trace how the Location value is handled when it contains Latin characters. Done means the reproduction no longer raises UnicodeDecodeError and the existing test suite passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.