airbytehq / airbytehq/airbyte

NPE in `MySqlGtidSet.UUIDSet.subtract` after RDS Blue/Green cutover (Debezium null-safety bug shipped in 3.53.x)

未关闭
#85,782 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
area/connectors autoteam community connectors/destination/s3 connectors/source/mysql needs-triage team/db-dw-sources team/use type/bug
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

### Connector Name

source-mysql

### Connector Version

3.53.3

### What step the error happened?

During the sync

### Relevant information

## Summary

After an AWS RDS Blue/Green promotion (MySQL 8.0 → 8.4), every CDC connection
against the promoted database fails during `validate()` with a raw
`NullPointerException` originating in Debezium's `MySqlGtidSet` code:

```
io.airbyte.cdk.ConfigErrorException: Incumbent CDC state is invalid, reason:
java.lang.NullPointerException: Cannot invoke
"io.debezium.connector.mysql.gtid.MySqlGtidSet$UUIDSet.getUUID()"
because "other" is null
at io.airbyte.cdk.read.cdc.CdcPartitionsCreator.run(CdcPartitionsCreator.kt:85)

```

The connector cannot self-recover. The user-facing message ("Incumbent CDC
state is invalid") gives no actionable signal — it hides the underlying
Debezium bug and looks identical to a legitimate "state is stale" error.

## Environment

- `airbyte/source-mysql:3.53.3`
- Destination: any (reproduced with `airbyte/destination-s3:1.9.8`)
- Source DB: AWS RDS for MySQL 8.4.9 (promoted from 8.0.42 via Blue/Green)
- GTID mode: enabled (default on RDS MySQL)

## Reproduction

Any operation that changes the server's `server_uuid` set will trigger it.
Easiest reproducers:

1. Set up a MySQL CDC connection against an RDS 8.0 instance. Let it sync.
2. Do an RDS Blue/Green promotion to 8.4 (or restore a snapshot into a new
cluster with different UUIDs). The blue writer's `server_uuid` will not
exist on the green side; the new green writer(s) will have UUIDs the saved
state has never seen.
3. Trigger a sync. `validate()` NPEs on the first `MySqlGtidSet.subtract` call
inside its `newGtidSet = availableGtidSet.subtract(savedGtidSet)` computation.
4. Even after manually stripping the vanished UUID from the saved state, the
NPE reappears from a different call site:
`newGtidSet.subtract(queryPurgedIds())`, because the green cluster has
UUIDs (typically the new writer's UUID) that don't appear in
`@@global.gtid_purged`.

## Root cause

The bug is upstream in Debezium and is already fixed on `main`, but Airbyte's
source-mysql 3.53.x pins an older Debezium that lacks the null check.

**Debezium `v2.7.4.Final` `MySqlGtidSet.UUIDSet.subtract`** (the version
effectively shipped inside `airbyte/source-mysql:3.53.3`):

```java
public UUIDSet subtract(UUIDSet other) {
if (!uuid.equals(other.getUUID())) { // NPE when other is null
throw new IllegalArgumentException(
"UUIDSet subtraction is supported only within a single server UUID");
}

}
```

Called from `MySqlGtidSet.subtract`:

```java
Map newSets = this.uuidSetsByServerId.entrySet()
.stream()
.filter(entry -> !entry.getValue().isContainedWithin(
theOther.forServerWithId(entry.getKey())))
.map(entry -> new AbstractMap.SimpleEntry<>(
entry.getKey(),
entry.getValue().subtract(theOther.forServerWithId(entry.getKey()))))

```

`forServerWithId` returns `null` for any UUID missing from the other side, and
`UUIDSet.subtract(null)` immediately dereferences `other.getUUID()`.

**Debezium `main` fixed this** — the first line of `UUIDSet.subtract` is now:

```java
public UUIDSet subtract(UUIDSet other) {
if (other == null) {
return this;
}
if (!uuid.equals(other.getUUID()) || !Objects.equals(tag, other.getTag())) {
throw new IllegalArgumentException(…);
}

}
```

`MySqlSourceDebeziumOperations.validate()` in the Airbyte source is otherwise
correct — it just has no way to defend itself when the underlying Debezium
subtract NPEs.

## Impact

- Every MySQL CDC connection against the migrated cluster fails identically.
- The only "official" recovery is a full connection reset, which triggers a
full re-snapshot of every table across every stream. On our fleet that was
~55 connections against a multi-TB database — hours of re-snapshot work and
destination churn to avoid ~20 lost transactions.
- The error message is misleading. Users see "Incumbent CDC state is invalid"
and typically follow the connector's own suggestion to "reset the connection
and increase binlog retention" — neither of which actually solves the NPE.

## Mitigation (what we did without upgrading the connector)

Because the connector will not self-recover, we manually patched each
connection's saved state before re-triggering the sync. The mitigation targets
both NPE call sites:

1. **Strip the vanished blue-writer UUID from every reference in the saved
state.** This removes the first NPE call site
(`availableGtidSet.subtract(savedGtidSet)`). Locations:
- `shared_state.state.mysql_cdc_offset[""].gtids`
- Every DDL event's `position.gtids` inside
`shared_state.state.mysql_db_history` (in our case ~433 events per
connection — the DDL snapshot from the initial connector bootstrap).
2. **Pad the saved GTID set with placeholder entries for every UUID present
on the green cluster that saved state has never seen.** This is what
defuses the second NPE (`newGtidSet.subtract(purgedGtidSet)`). Ranges must
be chosen carefully:
- For UUIDs present in `@@global.gtid_purged`: set the placeholder equal
to the purged upper bound. `newGtidSet` still contains
`[purged_end + 1, current_end]` for that UUID, so real transactions are
streamed, and `newGtidSet.subtract(purgedGtidSet)` produces
`newGtidSet` unchanged (validation passes).
- For UUIDs **not** in `@@global.gtid_purged`: set the placeholder equal
to the current `@@global.gtid_executed` range, so the UUID drops out of
`newGtidSet` entirely and the missing entry in `purgedGtidSet` is never
looked up. Any real events on this UUID would be lost, but in practice
the UUIDs that don't appear in `gtid_purged` on a fresh green cluster
are internal-only.

Example — from our saved offset before repair:

```
ac1cd931-37ce-11ef-a7f5-06e15e8c7d6f:1-3132,
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678
```

After repair (with `gtid_executed = 414620cc:1-12, b8232ea8:1-1932702885,
ca67b3cd:1-88121` and `gtid_purged = b8232ea8:1-1928281309, ca67b3cd:1-8`):

```
414620cc-aaa4-11f1-85e5-066c871909f5:1-12, # in executed, not in purged → pad to executed
ca67b3cd-aa9e-11f1-bb7c-06c0bbd01bd9:1-8, # in purged → pad to purged upper bound
b8232ea8-d1b6-11ee-a372-02dc0f37b73f:1-1932694678 # unchanged, real resume point
```

After this repair, `validate()` succeeds, Debezium resumes from the correct
GTID, and CDC catches up on the entire backlog with zero data gap (measured
against the pre-cutover parquet outputs). Total unrecoverable data loss
across the migration was 20 transactions on internal RDS setup UUIDs; zero
transactions on the primary data-carrying UUID.

## Requested fix

Any of the following would address the bug on the connector side:

1. **Bump the Debezium dependency** to a version that includes the
`UUIDSet.subtract(null)` null-safety fix. This is the cleanest solution.
2. **Wrap the two `MySqlGtidSet.subtract` calls in `validate()`** with a
`try { … } catch (NullPointerException npe) { … }` block that translates
the NPE into a `ConfigErrorException` with the actual failure mode
("saved state references a server UUID no longer present on the DB, this
usually indicates a Blue/Green cutover, restore-to-new-cluster, or replica
promotion — a connection reset is required unless you patch state
manually").
3. **Pre-check both directions of the UUID set before subtracting.** If saved
contains a UUID missing on server (or vice-versa in the purged direction),
raise a specific error rather than delegating to Debezium.

At minimum, the user-facing message should mention "Blue/Green cutover" or
"replica promotion" as a likely cause and should be distinguishable from a
routine "state is stale" error.

## Related

Debezium null-safety commit in `main`:
https://github.com/debezium/debezium/blob/main/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/gtid/MySqlGtidSet.java
(compare `UUIDSet.subtract` on `main` vs `v2.7.4.Final`). https://github.com/airbytehq/airbyte/issues/85783

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。