Cannot resume reading binlog from a position after 4GB [DBZ-2869]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-2869](https://issues.redhat.com/browse/DBZ-2869)
When requesting binlog from a position after 4GB (in this case, {{9886193806}}), MySQL tries to serve binlog from an earlier position ({{1296259214}}, which is exactly 8GB behind) and fails:
{noformat}
INFO: Connected to :3306 at mysql-bin-changelog.012139/9886193806 (sid:6401, cid:4394786)
org.apache.kafka.connect.errors.ConnectException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin-changelog.012139' at 1296259214, the last event read from '/path/to/binlog/mysql-bin-changelog.012139' at 185, the last byte read from '/path/to/binlog/mysql-bin-changelog.012139' at 1296259233.
{noformat}
Looks like the binlog reader [sends only the 4 lowest bytes|https://github.com/osheroff/mysql-binlog-connector-java/blob/c9c89037b398093e34af5b443b3bdb648bebb66c/src/main/java/com/github/shyiko/mysql/binlog/network/protocol/command/DumpBinaryLogCommand.java#L38-L41] of the position as part of {{COM_BINLOG_DUMP}}:
{code:java}
public byte[] toByteArray() throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
buffer.writeInteger(CommandType.BINLOG_DUMP.ordinal(), 1);
buffer.writeLong(this.binlogPosition, 4);
{code}
Which is a [limitation of the protocol|https://github.com/mysql/mysql-server/blob/e468c3634ba240cc00e83f6e4b1d7a8b31e2c292/sql/rpl_master.cc#L928-L933]:
{code:java}
/*
4 bytes is too little, but changing the protocol would break
compatibility. This has been fixed in the new protocol. @see
com_binlog_dump_gtid().
*/
READ_INT(pos, 4);
{code}
It's unclear whether this issue can lead to any data corruption but it should be accounted for. The options are:
# If the stored offset is larger than 4GB, instead of sending the 4 lower bytes, fail. It's extremely unlikely that just the 4 bytes represent a valid position to start reading from.
# If the stored offset is larger than 4GB, instead of sending the 4 lower bytes, resume from the latest known offset less than 4GB.
# See if {{com_binlog_dump_gtid}} could be used instead as suggested in the source code above.
Contributor guide
Research direction
Start with DumpBinaryLogCommand.java and the linked MySQL COM_BINLOG_DUMP protocol source to understand the 4-byte position limit. Reproduce or validate the failure with a stored position above 4GB, then assess the three proposed handling options, including whether com_binlog_dump_gtid can apply. Done means a decided safe behavior with coverage for positions beyond 4GB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100