latin-1 User-Agent is breaking requests
- Dominant language
- Python
- Stars
- 48
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
We have a user with a latin-1 encoded user agent:
```
>>> b"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3V\xE4xel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36".decode('utf-8')
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 83: invalid continuation byte
>>> b"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3V\xE4xel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36".decode('latin-1')
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 3Växel/1.5.0-239 Chrome/108.0.5359.215 Electron/22.3.8 Safari/537.36'
```
With kong 3.8 this is breaking as it seems to make the assumptions that headers are utf-8 encoded:
```
----------------------------------------, context: ngx.timer
Exception occurred during processing of request from , context: ngx.timer
Traceback (most recent call last):, context: ngx.timer
File "/usr/local/lib/python3.10/dist-packages/kong_pdk/listener.py", line 59, in handle, context: ngx.timer
File "msgpack/_unpacker.pyx", line 540, in msgpack._cmsgpack.Unpacker.__next__, context: ngx.timer
File "msgpack/_unpacker.pyx", line 463, in msgpack._cmsgpack.Unpacker._unpack, context: ngx.timer
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 83: invalid continuation byte, context: ngx.timer
----------------------------------------, context: ngx.timer
```
However this assumption is incorrect as the encoding of HTTP headers seems to be [ill-defined](https://stackoverflow.com/questions/4400678/what-character-encoding-should-i-use-for-a-http-header). I don't think that HTTP client is necessarily making an "incorrect" assumption by using latin-1. The PDK probably should just treat headers as `bytes`, or at the very least have a fallback mechanism to ascii on the `str.decode`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with kong_pdk/listener.py and the msgpack unpacking path shown in the traceback. Reproduce the failure using the latin-1 User-Agent example, then trace how header values are decoded. Done means the request can be processed without a UnicodeDecodeError while preserving the header value appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100