airbytehq / airbytehq/airbyte

[source-mysql] CDC fails on MySQL 8.4 tagged GTIDs: NumberFormatException in GtidSet parser

Offen
#85,846 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community connectors/source/mysql team/use
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Connector Name

source-mysql

## Connector Version

3.53.4

## What step the error happened?

During the sync

## Relevant information

### Summary

MySQL 8.4 supports **tagged GTIDs**, where a GTID set element is `uuid:[tag:]interval[:[tag:]interval]...`. The `GtidSet` parser bundled with the connector has no handling for the tag component: it splits everything after the first `:` on `:` and passes every resulting token to `Long.parseLong`. When the server's executed-GTID set contains a tag, parsing throws `NumberFormatException` and the sync fails as a `config_error`.

In our case the tag is `mysqlsh`, so the failure message is `NumberFormatException: For input string: "mysqlsh"`. Any legal tag reproduces it.

This is deterministic, not flaky — every subsequent sync on the affected connection fails identically. Because it is classified `config_error` rather than a transient error, there is no retry.

### Where it fails

The crash is in `MySqlConnection.knownGtidSet()`, i.e. while reading the GTID set **live from the server**, not while deserializing saved state:

```
BinlogStreamingChangeEventSource.execute(BinlogStreamingChangeEventSource.java:232)
-> MySqlConnection.knownGtidSet(MySqlConnection.java:71)
-> JdbcConnection.queryAndMap(...)
-> MySqlConnection.lambda$knownGtidSet$2(MySqlConnection.java:73)
-> MySqlGtidSet.(MySqlGtidSet.java:42)
-> GtidSet.(GtidSet.java:61) <-- NumberFormatException
```

Note this happens *before* any offset/position comparison and before GTID include/exclude filtering, so neither the "Invalid CDC Position Behavior" setting nor `gtid.source.excludes` can work around it.

### Root cause

MySQL 8.4 GTID set grammar ([docs](https://dev.mysql.com/doc/refman/8.4/en/replication-gtids-concepts.html)):

```
uuid_set: uuid:[tag:]interval[:[tag:]interval]...
tag: [a-zA-Z_][a-zA-Z0-9_]{0,31}
```

Manual's own example of a valid set: `3E11FA47-71CA-11E1-9E33-C80AA9429562:Domain_1:1-3:15-21:Domain_2:8-52`

`GtidSet` (`com.github.shyiko.mysql.binlog.GtidSet`, from osheroff/mysql-binlog-connector-java) parses it as:

```java
int uuidSeparatorIndex = uuidSet.indexOf(":");
UUID sourceId = UUID.fromString(uuidSet.substring(0, uuidSeparatorIndex));
String[] rawIntervals = uuidSet.substring(uuidSeparatorIndex + 1).split(":");
...
String[] is = interval.split("-");
split[i] = Long.parseLong(is[i]); // GtidSet.java:61
```

Every token after the first colon is assumed to be a numeric interval. There is no tag branch, so the tag token reaches `Long.parseLong` and throws.

### Steps to reproduce

1. MySQL 8.4 source with GTID mode enabled.
2. Ensure at least one tagged GTID exists in `@@global.gtid_executed` (e.g. any transaction committed under a tagged `gtid_next`; MySQL Shell / AdminAPI operations produce the `mysqlsh` tag). Verify with:
```sql
SELECT @@version, @@global.gtid_executed;
```
A tagged set looks like `:mysqlsh:1-42`.
3. Configure a source-mysql connection with Update Method = **Read Changes using Change Data Capture (CDC)**.
4. Run a sync. The snapshot phase succeeds; the streaming phase fails immediately.

### Expected

`GtidSet` parses tagged GTID sets per the MySQL 8.4 grammar, treating the tag as part of the UUID key as the server does.

### Actual

Sync fails with `config_error`.

### Relevant log output

```
failureOrigin: source
failureType: config_error
externalMessage: MySQL Connector Error: The sync encountered an unexpected error in the change event producer and has stopped.

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:150)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NumberFormatException: For input string: "mysqlsh"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at com.github.shyiko.mysql.binlog.GtidSet.(GtidSet.java:61)
at io.debezium.connector.mysql.gtid.MySqlGtidSet.(MySqlGtidSet.java:42)
at io.debezium.connector.mysql.jdbc.MySqlConnection.lambda$knownGtidSet$2(MySqlConnection.java:73)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:649)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:515)
at io.debezium.connector.mysql.jdbc.MySqlConnection.knownGtidSet(MySqlConnection.java:71)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource.execute(BinlogStreamingChangeEventSource.java:232)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource.execute(BinlogStreamingChangeEventSource.java:103)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:322)
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
... 5 more
```

### Environment

- source-mysql 3.53.4
- destination-snowflake 4.0.43
- container-orchestrator 2.0.1
- Airbyte on Kubernetes
- Update Method: CDC; Invalid CDC Position Behavior: Fail sync

### Notes

- Existing GTID issues appear to be distinct from this one: #85782 (NPE in `MySqlGtidSet.UUIDSet.subtract`), #82786 (null GTID in saved state), #75558 (`other is null` during GTID validation). None cover tag parsing.
- The defect is in the bundled binlog client rather than Airbyte code, so the fix may need to land upstream (Debezium / osheroff's mysql-binlog-connector-java) and be picked up via a dependency bump. I was not able to check the Debezium issue tracker to see whether it is already reported there.
- I inspected `GtidSet.java` at the fork's `master`, not the exact jar pinned in 3.53.4; the matching line number (61) suggests they correspond, but I did not diff them.

### Workaround

Switching the connection from CDC to "Scan Changes with User Defined Cursor" avoids the GTID path entirely, at the cost of CDC semantics (notably delete capture).

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.