julien-duponchelle / julien-duponchelle/python-mysql-replication

Decoding binary as utf-8

Open
#132 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

I get this trace:

```
Traceback (most recent call last):
File "./test.py", line 33, in
main()
File "./test.py", line 26, in main
for binlogevent in stream:
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/binlogstream.py", line 262, in fetchone
self.__freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/packet.py", line 98, in __init__
freeze_schema = freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/event.py", line 141, in __init__
self.query = tmp.decode("utf-8")
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xae in position 460: invalid start byte
```

I printed a repr of the packet and it is essentially:

```
INSERT INTO x(......, ip) VALUES (...., 'b\xae\xe1\xbd');
```

This is row based replication where the master was originally sent `INET6_ATON('::1')` for example.

What's the recommended solution here? I'm surprised no one else has hit this as many column types leverage binary.

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start at event.py line 141, where tmp is decoded as UTF-8, and trace how the packet is built in packet.py line 98 and consumed by binlogstream.py line 262. Reproduce the failure using the test.py example and inspect how binary values from row-based replication are represented. Done should include an agreed handling for binary data without the reported UnicodeDecodeError, backed by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.