UTF8 characters at 8192 byte boundary will cause errors
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
```
'utf-8' codec can't decode byte 0xc3 in position 8191: unexpected end of data: UnicodeDecodeError
Traceback (most recent call last):
File "/var/task/serverless_sdk/__init__.py", line 54, in wrapped_handler
return user_handler(event, context)
File "/var/task/handler.py", line 162, in slack
offers = get_lunch_offers(city)
File "/var/task/handler.py", line 100, in get_lunch_offers
offers = streaming_download_and_parse_offers(url)
File "/var/task/handler.py", line 89, in streaming_download_and_parse_offers
res_data = response.read(8192).decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 8191: unexpected end of data
```
My assumption is that there's a UTF8 character consisting of multiple bytes at the boundary of the byte array. The fix would be to read 8192 bytes, try decode, if fails, read 8192 mode and retry.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.