tornadoweb / tornadoweb/tornado
UnicodeDecodeError in curl_httpclient's `_curl_debug()`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22.2k
- Forks
- 5.6k
- Avg merge
- 3h 42m
- Merged PRs (30d)
- 16
Description
We ran into an issue in SaltStack with curl_httpclient when using a proxy and trying to download a binary file:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 1: invalid continuation byte
It seems, the fix for this issue is quite similar to the one for #1608 in d7d9c467cda38f4c9352172ba7411edc29a85196, but I'm not sure whether this would have also to be extended to the other cases a few lines below as 27a6103148a78100fc479f5dfc65065ef8e6ea72 changed the usage of native_str().
This hotfix (applied to the Tornado bundled with Salt) made it at least work for me and it seems .decode('latin1') isn't used in current master as well.
diff --git a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
index 8652343..6ef3349 100644
--- a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig
+++ b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
@@ -494,7 +494,7 @@ class CurlAsyncHTTPClient(AsyncHTTPClient):
def _curl_debug(self, debug_type, debug_msg):
debug_types = ('I', '<', '>', '<', '>')
- debug_msg = native_str(debug_msg)
+ debug_msg = native_str(debug_msg.decode('latin1'))
if debug_type == 0:
curl_log.debug('%s', debug_msg.strip())
elif debug_type in (1, 2):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in curl_httpclient.py at CurlAsyncHTTPClient._curl_debug and compare its native_str usage with the fix referenced in #1608. Inspect the other nearby native_str cases mentioned in the issue, then verify that proxy downloads of binary data no longer raise UnicodeDecodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100