NPE in `MySqlGtidSet.UUIDSet.subtract` after RDS Blue/Green cutover (Debezium null-safety bug shipped in 3.53.x)
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Connector Name
source-mysql
### Connector Version
3.53.3
### What step the error happened?
During the sync
### Relevant information
## Summary
After an AWS RDS Blue/Green promotion (MySQL 8.0 → 8.4), every CDC connection
against the promoted database fails during `validate()` with a raw
`NullPointerException` originating in Debezium's `MySqlGtidSet` code:
```
io.airbyte.cdk.ConfigErrorException: Incumbent CDC state is invalid, reason:
java.lang.NullPointerException: Cannot invoke
"io.debezium.connector.mysql.gtid.MySqlGtidSet$UUIDSet.getUUID()"
because "other" is null
at io.airbyte.cdk.read.cdc.CdcPartitionsCreator.run(CdcPartitionsCreator.kt:85)
…
```
The connector cannot self-recover. The user-facing message ("Incumbent CDC
state is invalid") gives no actionable signal — it hides the underlying
Debezium bug and looks identical to a legitimate "state is stale" error.
## Environment
- `airbyte/source-mysql:3.53.3`
- Destination: any (reproduced with `airbyte/destination-s3:1.9.8`)
- Source DB: AWS RDS for MySQL 8.4.9 (promoted from 8.0.42 via Blue/Green)
- GTID mode: enabled (default on RDS MySQL)
## Reproduction
Any operation that changes the server's `server_uuid` set will trigger it.
Easiest reproducers:
1. Set up a MySQL CDC connection against an RDS 8.0 instance. Let it sync.
2. Do an RDS Blue/Green promotion to 8.4 (or restore a snapshot into a new
cluster with different UUIDs). The blue writer's `server_uuid` will not
exist on the green side; the new green writer(s) will have UUIDs the saved
state has never seen.
3. Trigger a sync. `validate()` NPEs on the first `MySqlGtidSet.subtract` call
inside its `newGtidSet = availableGtidSet.subtract(savedGtidSet)` computation.
4. Even after manually stripping the vanished UUID from the saved state, the
NPE reappears from a different call site:
`newGtidSet.subtract(queryPurgedIds())`, because the green cluster has
UUIDs (typically the new writer's UUID) that don't appear in
`@@global.gtid_purged`.
## Root cause
The bug is upstream in Debezium and is already fixed on `main`, but Airbyte's
source-mysql 3.53.x pins an older Debezium that lacks the null check.
**Debezium `v2.7.4.Final` `MySqlGtidSet.UUIDSet.subtract`** (the version
effectively shipped inside `airbyte/source-mysql:3.53.3`):
```java
public UUIDSet subtract(UUIDSet other) {
if (!uuid.equals(other.getUUID())) { // NPE when other is null
throw new IllegalArgumentException(
"UUIDSet subtraction is supported only within a single server UUID");
}
…
}
```
Called from `MySqlGtidSet.subtract`:
```java
Map newSets = this.uuidSetsByServerId.entrySet()
.stream()
.filter(entry -> !entry.getValue().isContainedWithin(
theOther.forServerWithId(entry.getKey())))
.map(entry -> new AbstractMap.SimpleEntry<>(
entry.getKey(),
entry.getValue().subtract(theOther.forServerWithId(entry.getKey()))))
…
```
`forServerWithId` returns `null` for any UUID missing from the other side, and
`UUIDSet.subtract(null)` immediately dereferences `other.getUUID()`.
**Debezium `main` fixed this** — the first line of `UUIDSet.subtract` is now:
```java
public UUIDSet subtract(UUIDSet other) {
if (other == null) {
return this;
}
if (!uuid.equals(other.getUUID()) || !Objects.equals(tag, other.getTag())) {
throw new IllegalArgumentException(…);
}
…
}
```
`MySqlSourceDebeziumOperations.validate()` in the Airbyte source is otherwise
correct — it just has no way to defend itself when the underlying Debezium
subtract NPEs.
## Impact
- Every MySQL CDC connection against the migrated cluster fails identically.
- The only "official" recovery is a full connection reset, which triggers a
full re-snapshot of every table across every stream. On our fleet that was
~55 connections against a multi-TB database — hours of re-snapshot work and
destination churn to avoid ~20 lost transactions.
- The error message is misleading. Users see "Incumbent CDC state is invalid"
and typically follow the connector's own suggestion to "reset the connection
and increase binlog retention" — neither of which actually solves the NPE.
## Mitigation (what we did without upgrading the connector)
Because the connector will not self-recover, we manually patched each
connection's saved state before re-triggering the sync. The mitigation targets
both NPE call sites:
1. **Strip the vanished blue-writer UUID from every reference in the saved
state.** This removes the first NPE call site
(`availableGtidSet.subtract(savedGtidSet)`). Locations:
- `shared_state.state.mysql_cdc_offset[""].gtids`
- Every DDL event's `position.gtids` inside
`shared_state.state.mysql_db_history` (in our case ~433 events per
connection — the DDL snapshot from the initial connector bootstrap).
2. **Pad the saved GTID set with placeholder entries for every UUID present
on the green cluster that saved state has never seen.** This is what
defuses the second NPE (`newGtidSet.subtract(purgedGtidSet)`). Ranges must
be chosen carefully:
- For UUIDs present in `@@global.gtid_purged`: set the placeholder equal
to the purged upper bound. `newGtidSet` still contains
`[purged_end + 1, current_end]` for that UUID, so real transactions are
streamed, and `newGtidSet.subtract(purgedGtidSet)` produces
`newGtidSet` unchanged (validation passes).
- For UUIDs **not** in `@@global.gtid_purged`: set the placeholder equal
to the current `@@global.gtid_executed` range, so the UUID drops out of
`newGtidSet` entirely and the missing entry in `purgedGtidSet` is never
looked up. Any real events on this UUID would be lost, but in practice
the UUIDs that don't appear in `gtid_purged` on a fresh green cluster
are internal-only.
Example — from our saved offset before repair:
```
ac1cd931-37ce-11ef-a7f5-06e15e8c7d6f:1-3132,
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678
```
After repair (with `gtid_executed = 414620cc:1-12, b8232ea8:1-1932702885,
ca67b3cd:1-88121` and `gtid_purged = b8232ea8:1-1928281309, ca67b3cd:1-8`):
```
414620cc-aaa4-11f1-85e5-066c871909f5:1-12, # in executed, not in purged → pad to executed
ca67b3cd-aa9e-11f1-bb7c-06c0bbd01bd9:1-8, # in purged → pad to purged upper bound
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678 # unchanged, real resume point
```
After this repair, `validate()` succeeds, Debezium resumes from the correct
GTID, and CDC catches up on the entire backlog with zero data gap (measured
against the pre-cutover parquet outputs). Total unrecoverable data loss
across the migration was 20 transactions on internal RDS setup UUIDs; zero
transactions on the primary data-carrying UUID.
## Requested fix
Any of the following would address the bug on the connector side:
1. **Bump the Debezium dependency** to a version that includes the
`UUIDSet.subtract(null)` null-safety fix. This is the cleanest solution.
2. **Wrap the two `MySqlGtidSet.subtract` calls in `validate()`** with a
`try { … } catch (NullPointerException npe) { … }` block that translates
the NPE into a `ConfigErrorException` with the actual failure mode
("saved state references a server UUID no longer present on the DB, this
usually indicates a Blue/Green cutover, restore-to-new-cluster, or replica
promotion — a connection reset is required unless you patch state
manually").
3. **Pre-check both directions of the UUID set before subtracting.** If saved
contains a UUID missing on server (or vice-versa in the purged direction),
raise a specific error rather than delegating to Debezium.
At minimum, the user-facing message should mention "Blue/Green cutover" or
"replica promotion" as a likely cause and should be distinguishable from a
routine "state is stale" error.
## Related
Debezium null-safety commit in `main`:
https://github.com/debezium/debezium/blob/main/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/gtid/MySqlGtidSet.java
(compare `UUIDSet.subtract` on `main` vs `v2.7.4.Final`). https://github.com/airbytehq/airbyte/issues/85783
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
Guia de contribuição
Direção de pesquisa
Start at MySqlSourceDebeziumOperations.validate() and CdcPartitionsCreator.kt:85, then compare MySqlGtidSet.java UUIDSet.subtract in Debezium v2.7.4.Final with the linked main version. Trace both subtract calls for UUIDs missing from either set and determine which requested fix is appropriate. Done means Blue/Green or replica-promotion state no longer produces a raw NPE and the resulting validation behavior is distinguishable from ordinary stale state.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- aws, java, kotlin, mysql
- Domínio
- data-engineering, databases
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Ativa
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 48/100