Pylons / Pylons/webob

Encoding issue in request.py

Open
#464 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
443
Forks
206
PR merge metrics
No merged PRs in 30d

Description

I'm not entirely sure what the intent is here so hesitate to file a PR. We saw some errors thrown by our webapp (using gunicorn) and traced it to request.encget():

  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/webob/request.py", line 495, in url
    url = self.path_url
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/webob/request.py", line 467, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/webob/descriptors.py", line 70, in fget
    return req.encget(key, encattr=encattr)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/webob/request.py", line 165, in encget
    return bytes_(val, 'latin-1').decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 66: invalid start byte"

My read of util.byte_ is that, when passed a string, it performs val.encode() on it. So the following code in encget():

return bytes_(val, "latin-1").decode(encoding)

is the same as doing:

return val.encode("latin-1", "strict").decode(encoding)

Based on our exception we can see that the value of encoding is "utf-8", which gives us:

return val.encode("latin-1", "strict").decode("utf-8")

or with a specific example that will fail:

x = "À".encode('latin-1').decode('utf-8')

I'm not sure why we'd ever be explicitly encoding a string as latin-1 and then decoding it as UTF-8 in the first place -- a simpler return val.encode(encoding) would seem more appropriate here -- but again, there's probably nuance that I'm not understanding, hence the issue report.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in webob/request.py at encget(), then inspect util.byte_ as referenced in the report. Reproduce the failing "À" latin-1/UTF-8 example and determine the intended encoding behavior before changing it. Done means the behavior is clarified and the reported UnicodeDecodeError is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.