julien-duponchelle / julien-duponchelle/python-mysql-replication
Decoding binary as utf-8
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
I get this trace:
```
Traceback (most recent call last):
File "./test.py", line 33, in
main()
File "./test.py", line 26, in main
for binlogevent in stream:
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/binlogstream.py", line 262, in fetchone
self.__freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/packet.py", line 98, in __init__
freeze_schema = freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/event.py", line 141, in __init__
self.query = tmp.decode("utf-8")
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xae in position 460: invalid start byte
```
I printed a repr of the packet and it is essentially:
```
INSERT INTO x(......, ip) VALUES (...., 'b\xae\xe1\xbd');
```
This is row based replication where the master was originally sent `INET6_ATON('::1')` for example.
What's the recommended solution here? I'm surprised no one else has hit this as many column types leverage binary.
Thanks!
Contributor guide
Research direction
Start at event.py line 141, where tmp is decoded as UTF-8, and trace how the packet is built in packet.py line 98 and consumed by binlogstream.py line 262. Reproduce the failure using the test.py example and inspect how binary values from row-based replication are represented. Done should include an agreed handling for binary data without the reported UnicodeDecodeError, backed by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100