Behaviors are different when data is a list(dict) and text
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.5k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
When data is unicode, it gives traceback.
A simple test case:
request = requests.Request(method='GET', url='https://httpbin.org/get')
request.data = u'x=føø'
prepared = request.prepare()
requests.Session().send(prepared)
Traceback (most recent call last):
...
requests.Session().send(prepared)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 566, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 331, in send
timeout=timeout
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 558, in urlopen
body=body, headers=headers)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 975, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1009, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 971, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
self.send(message_body)
File "/usr/lib/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 329, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 298, in send
v = self._sslobj.write(data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: ordinal not in range(128)
When data is a list, it works fine.
request = requests.Request(method='GET', url='https://httpbin.org/get')
request.data = [(u'x', u'føø')]
prepared = request.prepare()
requests.Session().send(prepared)
Requests seems to be supporting unicode data, shown in the code. However, it seems not work well, and the behavior differs when the data is a list-like type and a text-like type, one try to encode using 'uft-8' and the other just pass through(in _encode_params).
def prepare_body(self, data, files, json=None):
...
if files:
(body, content_type) = self._encode_files(files, data)
else:
if data and json is None:
body = self._encode_params(data) #===> check data
if isinstance(data, basestring) or hasattr(data, 'read'):
content_type = None
else:
content_type = 'application/x-www-form-urlencoded'
...
@staticmethod
def _encode_params(data):
if isinstance(data, (str, bytes)): #===> allows unicode
return data #===> simply return
elif hasattr(data, 'read'):
return data
elif hasattr(data, '__iter__'): #===> the behavior is different.
result = []
for k, vs in to_key_val_list(data):
if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):
vs = [vs]
for v in vs:
if v is not None:
result.append(
(k.encode('utf-8') if isinstance(k, str) else k,
v.encode('utf-8') if isinstance(v, str) else v)) #===>you try to encode as 'utf8'
return urlencode(result, doseq=True)
else:
return data
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Request.prepare_body and _encode_params, then reproduce the reported Python 2 behavior using the scalar Unicode and list-of-tuples examples. Compare how each input is encoded and sent, and add regression coverage for the chosen consistent behavior. Done means Unicode request data no longer raises the reported encoding error and the existing list form remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100