tornadoweb / tornadoweb/tornado

UnicodeDecodeError in curl_httpclient's `_curl_debug()`

Open
#3,183 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

httpclient
Dominant language
Python
Stars
22.2k
Forks
5.6k
Avg merge
3h 42m
Merged PRs (30d)
16

Description

We ran into an issue in SaltStack with curl_httpclient when using a proxy and trying to download a binary file:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 1: invalid continuation byte

It seems, the fix for this issue is quite similar to the one for #1608 in d7d9c467cda38f4c9352172ba7411edc29a85196, but I'm not sure whether this would have also to be extended to the other cases a few lines below as 27a6103148a78100fc479f5dfc65065ef8e6ea72 changed the usage of native_str().

This hotfix (applied to the Tornado bundled with Salt) made it at least work for me and it seems .decode('latin1') isn't used in current master as well.

diff --git a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
index 8652343..6ef3349 100644
--- a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig
+++ b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
@@ -494,7 +494,7 @@ class CurlAsyncHTTPClient(AsyncHTTPClient):

     def _curl_debug(self, debug_type, debug_msg):
         debug_types = ('I', '<', '>', '<', '>')
-        debug_msg = native_str(debug_msg)
+        debug_msg = native_str(debug_msg.decode('latin1'))
         if debug_type == 0:
             curl_log.debug('%s', debug_msg.strip())
         elif debug_type in (1, 2):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in curl_httpclient.py at CurlAsyncHTTPClient._curl_debug and compare its native_str usage with the fix referenced in #1608. Inspect the other nearby native_str cases mentioned in the issue, then verify that proxy downloads of binary data no longer raise UnicodeDecodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.