psf / psf/requests

Unicode surrogates in POST data lead to exception

Open
#6,008 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

In requests 2.26.0 with Python 3, when passing unicode strings with surrogate characters as POST data (either directly as a str or as part of a dict), sending the request throws an exception.

Expected Result

The library should not throw an exception, even when fed with unvalidated user input.

Actual Result / Reproduction Steps

>>> import requests
>>> requests.__version__
'2.26.0'

>>> requests.post('https://example.invalid', data={'name': 'test\udced\udcb3\udc83.pdf'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.7/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/home/user/.local/lib/python3.7/site-packages/requests/sessions.py", line 466, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/home/user/.local/lib/python3.7/site-packages/requests/models.py", line 319, in prepare
    self.prepare_body(data, files, json)
  File "/home/user/.local/lib/python3.7/site-packages/requests/models.py", line 515, in prepare_body
    body = self._encode_params(data)
  File "/home/user/.local/lib/python3.7/site-packages/requests/models.py", line 104, in _encode_params
    v.encode('utf-8') if isinstance(v, str) else v))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 4-6: surrogates not allowed

>>> requests.post('https://example.com/', 'test\udced\udcb3\udc83.pdf')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.7/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/user/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/user/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/user/.local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.7/http/client.py", line 1260, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1305, in _send_request
    body = _encode(body, 'body')
  File "/usr/lib/python3.7/http/client.py", line 174, in _encode
    (name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 4-6: Body ('\udced\udcb3\udc83') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.9"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "4.19.0-17-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

In requests/models.py, there are several calls like some_string.encode('utf-8'). I suppose changing them to some_string.encode('utf-8', 'surrogatepass') would at least fix the first reproduction case. Not sure about the second, since the exception occurs in http.client there.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in requests/models.py, especially prepare_body and _encode_params, then reproduce both POST examples from the issue. Trace where each surrogate-containing value is encoded, including the path that reaches Python's http.client. Done means both forms no longer raise an exception for the reported input, with regression coverage for the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.