julien-duponchelle / julien-duponchelle/python-mysql-replication
Possibly incomplete support of CRC32 checksums
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
This is relating to #4 :
We have an AWS MySQL Aurora server we are reading from using this great tool. The server has CRC32 checksum turned on, and most events are read properly. However, once in a while, the Rotate event at the end of a logfile does not contain the 4 checksum bytes at the end. This causes the library to improperly read the RotateEvent, ignoring the last 4 bytes of the next binlog name:
This packet, for instance:
``` Python
>>> x = m.fetchone()
>>> x.packet.packet._data
'\x00\x95\xda\xc7Z\x04\x917_\x0b5\x00\x00\x00\x0bd\xb3\x03\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00mysql-bin-changelog.000049'
>>> x.next_binlog
u'mysql-bin-changelog.00'
```
As you can see, the packet has no checksum - it ends with the bin log file name.
I went ahead and checked the underlying connection and the next recv returns the next packet (meaning the packet itself seems to have been read properly - it just doesn't contain a checksum.
It looks like `mysqlbinlog` reads this packet properly, which lead me to believe that this is either a bug in Aurora that causes _some_ packets to be sent w/o a checksum and the implementation of the MySQL protocol doesn't fail on it, or that this is in fact a legitimate packet structure.
The problem root cause in pymysqlreplication is here (packet.py):
``` Python
# MySQL 5.6 and more if binlog-checksum = CRC32
if use_checksum:
event_size_without_header = self.event_size - 23
else:
event_size_without_header = self.event_size - 19
```
The code assumes that the event size should be decreased by 4 when checksum is enabled, but in this case, it shouldn't be. I haven't had time to figure out the best solution yet, so mainly raising awareness and igniting a discussion.
Contributor guide
Research direction
Start in packet.py at the event-size calculation shown in the issue, then inspect how RotateEvent consumes the packet and how checksum-enabled events are distinguished. Reproduce the Aurora packet or compare its handling with mysqlbinlog; done means preserving the full binlog filename without breaking ordinary CRC32 events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100