psf / psf/requests

Behaviors are different when data is a list(dict) and text

Open
#2,638 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.5k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

When data is unicode, it gives traceback.
A simple test case:

request = requests.Request(method='GET', url='https://httpbin.org/get')
request.data =  u'x=føø'
prepared = request.prepare()
requests.Session().send(prepared)
Traceback (most recent call last):
...
    requests.Session().send(prepared)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 566, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 331, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 558, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 975, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1009, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 971, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 329, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 298, in send
    v = self._sslobj.write(data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: ordinal not in range(128)

When data is a list, it works fine.

    request = requests.Request(method='GET', url='https://httpbin.org/get')
    request.data = [(u'x', u'føø')]
    prepared = request.prepare()
    requests.Session().send(prepared)

Requests seems to be supporting unicode data, shown in the code. However, it seems not work well, and the behavior differs when the data is a list-like type and a text-like type, one try to encode using 'uft-8' and the other just pass through(in _encode_params).

    def prepare_body(self, data, files, json=None):
            ...
            if files:
                (body, content_type) = self._encode_files(files, data)
            else:
                if data and json is None:
                    body = self._encode_params(data)     #===> check data
                    if isinstance(data, basestring) or hasattr(data, 'read'):
                        content_type = None
                    else:
                        content_type = 'application/x-www-form-urlencoded'
                    ...
    @staticmethod
    def _encode_params(data):
        if isinstance(data, (str, bytes)):   #===> allows unicode
            return data                              #===> simply return
        elif hasattr(data, 'read'):
            return data
        elif hasattr(data, '__iter__'):        #===> the behavior is different.
            result = []
            for k, vs in to_key_val_list(data):
                if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):
                    vs = [vs]
                for v in vs:
                    if v is not None:
                        result.append(
                            (k.encode('utf-8') if isinstance(k, str) else k,
                             v.encode('utf-8') if isinstance(v, str) else v))   #===>you try to encode as 'utf8'
            return urlencode(result, doseq=True)
        else:
            return data

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Request.prepare_body and _encode_params, then reproduce the reported Python 2 behavior using the scalar Unicode and list-of-tuples examples. Compare how each input is encoded and sent, and add regression coverage for the chosen consistent behavior. Done means Unicode request data no longer raises the reported encoding error and the existing list form remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.