Kong / Kong/kong-python-pdk

latin-1 User-Agent is breaking requests

Open
#146 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
48
Forks
16
PR merge metrics
No merged PRs in 30d

Description

We have a user with a latin-1 encoded user agent:

```
>>> b"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3V\xE4xel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36".decode('utf-8')
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 83: invalid continuation byte
>>> b"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3V\xE4xel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36".decode('latin-1')
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3Växel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36'
```

With kong 3.8 this is breaking as it seems to make the assumptions that headers are utf-8 encoded:

```
----------------------------------------, context: ngx.timer
Exception occurred during processing of request from , context: ngx.timer
Traceback (most recent call last):, context: ngx.timer
File "/usr/local/lib/python3.10/dist-packages/kong_pdk/listener.py", line 59, in handle, context: ngx.timer
File "msgpack/_unpacker.pyx", line 540, in msgpack._cmsgpack.Unpacker.__next__, context: ngx.timer
File "msgpack/_unpacker.pyx", line 463, in msgpack._cmsgpack.Unpacker._unpack, context: ngx.timer
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 83: invalid continuation byte, context: ngx.timer
----------------------------------------, context: ngx.timer
```

However this assumption is incorrect as the encoding of HTTP headers seems to be [ill-defined](https://stackoverflow.com/questions/4400678/what-character-encoding-should-i-use-for-a-http-header). I don't think that HTTP client is necessarily making an "incorrect" assumption by using latin-1. The PDK probably should just treat headers as `bytes`, or at the very least have a fallback mechanism to ascii on the `str.decode`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with kong_pdk/listener.py and the msgpack unpacking path shown in the traceback. Reproduce the failure using the latin-1 User-Agent example, then trace how header values are decoded. Done means the request can be processed without a UnicodeDecodeError while preserving the header value appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.