UnicodeDecodeError after following a chain of redirects

#6006

Something confuses requests (or urllib3?) along the way

Actual Result
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 92: invalid continuation byte

Reproduction Steps
import requests
requests.get("https://www.lavozdegalicia.es/noticia/deportes/2021/12/13/psg-juve-united-nuevos-rivales-espa%C3%B1oles-champions/00031639396272418389372.htm")

System Information
$ python -m requests.help
{
"chardet": {
"version": "4.0.0"
},
"charset_normalizer": {
"version": "2.0.9"
},
"cryptography": {
"version": "36.0.0"
},
"idna": {
"version": "3.3"
},
"implementation": {
"name": "CPython",
"version": "3.8.10"
},
"platform": {
"release": "4.4.0-17763-Microsoft",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "101010cf",
"version": "21.0.0"
},
"requests": {
"version": "2.26.0"
},
"system_ssl": {
"version": "1010106f"
},
"urllib3": {
"version": "1.26.7"
},
"using_charset_normalizer": false,
"using_pyopenssl": true
}

@sigmavirus24 you have been too harsh on this one.

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 677, in send
history = [resp for resp in gen]
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 677, in
history = [resp for resp in gen]
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 150, in resolve_redirects
url = self.get_redirect_target(resp)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 116, in get_redirect_target
return to_native_string(location, 'utf8')
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests_internal_utils.py", line 25, in to_native_string
out = string.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 92: invalid continuation byte

The guilty part there is return to_native_string(location, 'utf8') Trying to decode the URL to native utf8 when it should rather "URL encode" it. I am not an HTTP expert, but this exception should be handled more gracefully anyway.

The location given by the remote for redirection is as follow:

b'http://www.lavozdegalicia.es/noticia/deportes/2021/12/13/psg-juve-united-nuevos-rivales-espa\xf1oles-champions/00031639396272418389372.htm'

Should the \xf1 be decoded as follow %F1 ?

I can see that this behavior is already followed by Chrome.

Start with requests/sessions.py at get_redirect_target and requests_internal_utils.py at to_native_string, then run the provided requests.get reproduction URL. Trace how the redirect Location bytes are decoded and determine the expected handling for the shown non-UTF-8 byte. Done means the reproduction no longer raises UnicodeDecodeError and the redirect is handled consistently.