debezium / debezium/dbz

Cannot resume reading binlog from a position after 4GB [DBZ-2869]

Open
#401 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2869](https://issues.redhat.com/browse/DBZ-2869)

When requesting binlog from a position after 4GB (in this case, {{9886193806}}), MySQL tries to serve binlog from an earlier position ({{1296259214}}, which is exactly 8GB behind) and fails:
{noformat}
INFO: Connected to :3306 at mysql-bin-changelog.012139/9886193806 (sid:6401, cid:4394786)
org.apache.kafka.connect.errors.ConnectException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin-changelog.012139' at 1296259214, the last event read from '/path/to/binlog/mysql-bin-changelog.012139' at 185, the last byte read from '/path/to/binlog/mysql-bin-changelog.012139' at 1296259233.
{noformat}
Looks like the binlog reader [sends only the 4 lowest bytes|https://github.com/osheroff/mysql-binlog-connector-java/blob/c9c89037b398093e34af5b443b3bdb648bebb66c/src/main/java/com/github/shyiko/mysql/binlog/network/protocol/command/DumpBinaryLogCommand.java#L38-L41] of the position as part of {{COM_BINLOG_DUMP}}:
{code:java}
public byte[] toByteArray() throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
buffer.writeInteger(CommandType.BINLOG_DUMP.ordinal(), 1);
buffer.writeLong(this.binlogPosition, 4);
{code}
Which is a [limitation of the protocol|https://github.com/mysql/mysql-server/blob/e468c3634ba240cc00e83f6e4b1d7a8b31e2c292/sql/rpl_master.cc#L928-L933]:
{code:java}
/*
  4 bytes is too little, but changing the protocol would break
compatibility. This has been fixed in the new protocol. @see
com_binlog_dump_gtid().
*/
READ_INT(pos, 4);
{code}
It's unclear whether this issue can lead to any data corruption but it should be accounted for. The options are:
# If the stored offset is larger than 4GB, instead of sending the 4 lower bytes, fail. It's extremely unlikely that just the 4 bytes represent a valid position to start reading from.
# If the stored offset is larger than 4GB, instead of sending the 4 lower bytes, resume from the latest known offset less than 4GB.
# See if {{com_binlog_dump_gtid}} could be used instead as suggested in the source code above.

Contributor guide

Open the contributing guide

Research direction

Start with DumpBinaryLogCommand.java and the linked MySQL COM_BINLOG_DUMP protocol source to understand the 4-byte position limit. Reproduce or validate the failure with a stored position above 4GB, then assess the three proposed handling options, including whether com_binlog_dump_gtid can apply. Done means a decided safe behavior with coverage for positions beyond 4GB.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.