julien-duponchelle / julien-duponchelle/python-mysql-replication

Possibly incomplete support of CRC32 checksums

Open
#260 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

This is relating to #4 :

We have an AWS MySQL Aurora server we are reading from using this great tool. The server has CRC32 checksum turned on, and most events are read properly. However, once in a while, the Rotate event at the end of a logfile does not contain the 4 checksum bytes at the end. This causes the library to improperly read the RotateEvent, ignoring the last 4 bytes of the next binlog name:

This packet, for instance:
``` Python
>>> x = m.fetchone()
>>> x.packet.packet._data
'\x00\x95\xda\xc7Z\x04\x917_\x0b5\x00\x00\x00\x0bd\xb3\x03\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00mysql-bin-changelog.000049'
>>> x.next_binlog
u'mysql-bin-changelog.00'
```
As you can see, the packet has no checksum - it ends with the bin log file name.
I went ahead and checked the underlying connection and the next recv returns the next packet (meaning the packet itself seems to have been read properly - it just doesn't contain a checksum.

It looks like `mysqlbinlog` reads this packet properly, which lead me to believe that this is either a bug in Aurora that causes _some_ packets to be sent w/o a checksum and the implementation of the MySQL protocol doesn't fail on it, or that this is in fact a legitimate packet structure.

The problem root cause in pymysqlreplication is here (packet.py):
``` Python
# MySQL 5.6 and more if binlog-checksum = CRC32
if use_checksum:
event_size_without_header = self.event_size - 23
else:
event_size_without_header = self.event_size - 19
```

The code assumes that the event size should be decreased by 4 when checksum is enabled, but in this case, it shouldn't be. I haven't had time to figure out the best solution yet, so mainly raising awareness and igniting a discussion.

Contributor guide

Open the contributing guide

Research direction

Start in packet.py at the event-size calculation shown in the issue, then inspect how RotateEvent consumes the packet and how checksum-enabled events are distinguished. Reproduce the Aurora packet or compare its handling with mysqlbinlog; done means preserving the full binlog filename without breaking ordinary CRC32 events.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.