NPE in `MySqlGtidSet.UUIDSet.subtract` after RDS Blue/Green cutover (Debezium null-safety bug shipped in 3.53.x)
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
### Connector Name
source-mysql
### Connector Version
3.53.3
### What step the error happened?
During the sync
### Relevant information
## Summary
After an AWS RDS Blue/Green promotion (MySQL 8.0 → 8.4), every CDC connection
against the promoted database fails during `validate()` with a raw
`NullPointerException` originating in Debezium's `MySqlGtidSet` code:
```
io.airbyte.cdk.ConfigErrorException: Incumbent CDC state is invalid, reason:
java.lang.NullPointerException: Cannot invoke
"io.debezium.connector.mysql.gtid.MySqlGtidSet$UUIDSet.getUUID()"
because "other" is null
at io.airbyte.cdk.read.cdc.CdcPartitionsCreator.run(CdcPartitionsCreator.kt:85)
…
```
The connector cannot self-recover. The user-facing message ("Incumbent CDC
state is invalid") gives no actionable signal — it hides the underlying
Debezium bug and looks identical to a legitimate "state is stale" error.
## Environment
- `airbyte/source-mysql:3.53.3`
- Destination: any (reproduced with `airbyte/destination-s3:1.9.8`)
- Source DB: AWS RDS for MySQL 8.4.9 (promoted from 8.0.42 via Blue/Green)
- GTID mode: enabled (default on RDS MySQL)
## Reproduction
Any operation that changes the server's `server_uuid` set will trigger it.
Easiest reproducers:
1. Set up a MySQL CDC connection against an RDS 8.0 instance. Let it sync.
2. Do an RDS Blue/Green promotion to 8.4 (or restore a snapshot into a new
cluster with different UUIDs). The blue writer's `server_uuid` will not
exist on the green side; the new green writer(s) will have UUIDs the saved
state has never seen.
3. Trigger a sync. `validate()` NPEs on the first `MySqlGtidSet.subtract` call
inside its `newGtidSet = availableGtidSet.subtract(savedGtidSet)` computation.
4. Even after manually stripping the vanished UUID from the saved state, the
NPE reappears from a different call site:
`newGtidSet.subtract(queryPurgedIds())`, because the green cluster has
UUIDs (typically the new writer's UUID) that don't appear in
`@@global.gtid_purged`.
## Root cause
The bug is upstream in Debezium and is already fixed on `main`, but Airbyte's
source-mysql 3.53.x pins an older Debezium that lacks the null check.
**Debezium `v2.7.4.Final` `MySqlGtidSet.UUIDSet.subtract`** (the version
effectively shipped inside `airbyte/source-mysql:3.53.3`):
```java
public UUIDSet subtract(UUIDSet other) {
if (!uuid.equals(other.getUUID())) { // NPE when other is null
throw new IllegalArgumentException(
"UUIDSet subtraction is supported only within a single server UUID");
}
…
}
```
Called from `MySqlGtidSet.subtract`:
```java
Map newSets = this.uuidSetsByServerId.entrySet()
.stream()
.filter(entry -> !entry.getValue().isContainedWithin(
theOther.forServerWithId(entry.getKey())))
.map(entry -> new AbstractMap.SimpleEntry<>(
entry.getKey(),
entry.getValue().subtract(theOther.forServerWithId(entry.getKey()))))
…
```
`forServerWithId` returns `null` for any UUID missing from the other side, and
`UUIDSet.subtract(null)` immediately dereferences `other.getUUID()`.
**Debezium `main` fixed this** — the first line of `UUIDSet.subtract` is now:
```java
public UUIDSet subtract(UUIDSet other) {
if (other == null) {
return this;
}
if (!uuid.equals(other.getUUID()) || !Objects.equals(tag, other.getTag())) {
throw new IllegalArgumentException(…);
}
…
}
```
`MySqlSourceDebeziumOperations.validate()` in the Airbyte source is otherwise
correct — it just has no way to defend itself when the underlying Debezium
subtract NPEs.
## Impact
- Every MySQL CDC connection against the migrated cluster fails identically.
- The only "official" recovery is a full connection reset, which triggers a
full re-snapshot of every table across every stream. On our fleet that was
~55 connections against a multi-TB database — hours of re-snapshot work and
destination churn to avoid ~20 lost transactions.
- The error message is misleading. Users see "Incumbent CDC state is invalid"
and typically follow the connector's own suggestion to "reset the connection
and increase binlog retention" — neither of which actually solves the NPE.
## Mitigation (what we did without upgrading the connector)
Because the connector will not self-recover, we manually patched each
connection's saved state before re-triggering the sync. The mitigation targets
both NPE call sites:
1. **Strip the vanished blue-writer UUID from every reference in the saved
state.** This removes the first NPE call site
(`availableGtidSet.subtract(savedGtidSet)`). Locations:
- `shared_state.state.mysql_cdc_offset[""].gtids`
- Every DDL event's `position.gtids` inside
`shared_state.state.mysql_db_history` (in our case ~433 events per
connection — the DDL snapshot from the initial connector bootstrap).
2. **Pad the saved GTID set with placeholder entries for every UUID present
on the green cluster that saved state has never seen.** This is what
defuses the second NPE (`newGtidSet.subtract(purgedGtidSet)`). Ranges must
be chosen carefully:
- For UUIDs present in `@@global.gtid_purged`: set the placeholder equal
to the purged upper bound. `newGtidSet` still contains
`[purged_end + 1, current_end]` for that UUID, so real transactions are
streamed, and `newGtidSet.subtract(purgedGtidSet)` produces
`newGtidSet` unchanged (validation passes).
- For UUIDs **not** in `@@global.gtid_purged`: set the placeholder equal
to the current `@@global.gtid_executed` range, so the UUID drops out of
`newGtidSet` entirely and the missing entry in `purgedGtidSet` is never
looked up. Any real events on this UUID would be lost, but in practice
the UUIDs that don't appear in `gtid_purged` on a fresh green cluster
are internal-only.
Example — from our saved offset before repair:
```
ac1cd931-37ce-11ef-a7f5-06e15e8c7d6f:1-3132,
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678
```
After repair (with `gtid_executed = 414620cc:1-12, b8232ea8:1-1932702885,
ca67b3cd:1-88121` and `gtid_purged = b8232ea8:1-1928281309, ca67b3cd:1-8`):
```
414620cc-aaa4-11f1-85e5-066c871909f5:1-12, # in executed, not in purged → pad to executed
ca67b3cd-aa9e-11f1-bb7c-06c0bbd01bd9:1-8, # in purged → pad to purged upper bound
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678 # unchanged, real resume point
```
After this repair, `validate()` succeeds, Debezium resumes from the correct
GTID, and CDC catches up on the entire backlog with zero data gap (measured
against the pre-cutover parquet outputs). Total unrecoverable data loss
across the migration was 20 transactions on internal RDS setup UUIDs; zero
transactions on the primary data-carrying UUID.
## Requested fix
Any of the following would address the bug on the connector side:
1. **Bump the Debezium dependency** to a version that includes the
`UUIDSet.subtract(null)` null-safety fix. This is the cleanest solution.
2. **Wrap the two `MySqlGtidSet.subtract` calls in `validate()`** with a
`try { … } catch (NullPointerException npe) { … }` block that translates
the NPE into a `ConfigErrorException` with the actual failure mode
("saved state references a server UUID no longer present on the DB, this
usually indicates a Blue/Green cutover, restore-to-new-cluster, or replica
promotion — a connection reset is required unless you patch state
manually").
3. **Pre-check both directions of the UUID set before subtracting.** If saved
contains a UUID missing on server (or vice-versa in the purged direction),
raise a specific error rather than delegating to Debezium.
At minimum, the user-facing message should mention "Blue/Green cutover" or
"replica promotion" as a likely cause and should be distinguishable from a
routine "state is stale" error.
## Related
Debezium null-safety commit in `main`:
https://github.com/debezium/debezium/blob/main/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/gtid/MySqlGtidSet.java
(compare `UUIDSet.subtract` on `main` vs `v2.7.4.Final`). https://github.com/airbytehq/airbyte/issues/85783
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
贡献指南
评估
这个 Issue 还没有评估数据。