debezium / debezium/dbz

MySQL connector cannot restart gracefully when `GTID_MODE=ON_PERMISSIVE`: the offset's GTID is never maintained but is still validated

Open
#2,519 1 comment 0 reactions 0 assignees View on GitHub
component/mysql-connector type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

**What Debezium connector do you use and what version?**

MySQL connector. Observed on 3.0.7.Final. The relevant code is unchanged on 3.2.6.Final, 3.6.1.Final, 3.7.0.Beta1 and `main` (3.7.0-SNAPSHOT) - verified by inspection, see below.

---

**What is the connector configuration?**

Reduced to the properties that matter (host names, topic prefixes and credentials removed):

```json
{
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"snapshot.mode": "schema_only",
"read.only": "false",
"topic.prefix": "",
"database.hostname": "",
"database.port": "3306",
"incremental.snapshot.allow.schema.changes": "true",
"schema.history.internal.store.only.captured.tables.ddl": "false",
"heartbeat.interval.ms": "60000"
}
```

No `gtid.source.includes` / `gtid.source.excludes` are set.

---

**What is the captured database version and mode of deployment?**

_E.g. on-premises, with a specific cloud provider, etc._

Amazon Aurora MySQL, provisioned cluster, connected to the writer instance.

The distinguishing property is the server's GTID mode:

```sql
SELECT @@global.gtid_mode; -- ON_PERMISSIVE
```

Binlog retention is 168 hours (`binlog retention hours` = 168), with roughly 6 days actually retained.

---

**What behavior do you expect?**

The connector should restart successfully whenever the binlog file and position recorded in the offset are still available on the server. Under `GTID_MODE=ON_PERMISSIVE` the connector resumes streaming by binlog file/position, so that is the position whose availability determines whether a restart can succeed.

---

**What behavior do you see?**

Every restart fails once binlog retention has elapsed, even though the binlog file and position in the offset are current:

```
io.debezium.DebeziumException: The connector is trying to read change stream starting at
BinlogOffsetContext{sourceInfo=BinlogSourceInfo{currentGtid='null',
currentBinlogFilename='mysql-bin-changelog.210661', currentBinlogPosition=97623792, ...},
restartGtidSet='52dfc80f-df0d-3b9b-905b-4eb20dad19a7:1-2191466007',
currentGtidSet='52dfc80f-df0d-3b9b-905b-4eb20dad19a7:1-2191466007',
restartBinlogFilename='mysql-bin-changelog.210661', restartBinlogPosition=97623792, ...},
but this is no longer available on the server. Reconfigure the connector to use a snapshot mode when needed.
at io.debezium.connector.common.BaseSourceTask.validateSchemaHistory(BaseSourceTask.java:157)
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:138)
```

#### Root cause

`MySqlConnection.isGtidModeEnabled()` is looking for an exact match `"ON"`:

```java
// debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/jdbc/MySqlConnection.java
return queryAndMap("SHOW GLOBAL VARIABLES LIKE 'GTID_MODE'", rs -> {
if (rs.next()) {
return "ON".equalsIgnoreCase(rs.getString(2));
}
return false;
});
```

MySQL has four values for `gtid_mode`: `OFF`, `OFF_PERMISSIVE`, `ON_PERMISSIVE`, `ON`. Under `ON_PERMISSIVE` the server **does** assign a GTID to every new transaction and `gtid_executed` / `gtid_purged` grow normally; the mode only means the server additionally allows anonymous transactions arriving from a replication source. But the exact match reports `isGtidModeEnabled() == false`.

Three code paths then disagree about the offset's GTID field:

| Path | Behaviour |
|---|---|
| snapshot - `MySqlSnapshotChangeEventSource` | writes `SHOW MASTER STATUS` `Executed_Gtid_Set` into the offset **unconditionally** |
| streaming - `BinlogStreamingChangeEventSource` | registers the GTID event handler **only when `isGtidModeEnabled()`** |
| startup validation - `BinlogConnectorConnection.isBinlogPositionAvailable` | trusts the offset's GTID **unconditionally** (`if (gtid != null)`) |

`handleGtidEvent` is the only code that advances the offset's GTID during streaming. With no handler registered, the GTID written by the snapshot never changes, while `file`, `pos` and `ts_sec` advance normally. Both live in the same offset, which makes the divergence visible - from `Stopped reading binlog` at the moment of a clean shutdown:

```
Stopped reading binlog after 167693246 events, last recorded offset:
{ts_sec=1787677593, file=mysql-bin-changelog.210661, pos=97701859,
gtids=52dfc80f-df0d-3b9b-905b-4eb20dad19a7:1-2191466007, server_id=..., event=1}
```

`ts_sec` is the shutdown second and `file`/`pos` are at the head of the binlog, but `gtids` is 8 days stale.

`isBinlogPositionAvailable` enters its GTID branch whenever the offset carries a non-empty GTID, never reaches the binlog-file check, and compares that frozen snapshot-time watermark against a moving `gtid_purged`:

```
GTID Set retained: '52dfc80f-...:1-2191466007'
Server has already purged '52dfc80f-...:1-2234217820' GTIDs
GTIDs known by the server but not processed yet '52dfc80f-...:2191466008-2468638141',
for replication are available only '52dfc80f-...:2234217821-2468638141'
Some of the GTIDs needed to replicate have been already purged
```

So the connector refuses to start on the grounds that data has been purged, while sitting at the head of a fully available binlog. The pattern is: works for about six days after a fresh snapshot or offset reset, then every restart is fatal until the offsets are deleted, which repeats the issue.

Note also `currentGtid='null'` in the exception alongside a fully populated `restartGtidSet`. `currentGtid` is set only by `startGtid()` inside `handleGtidEvent`, so its being null is direct evidence that the handler never ran.

---

### Do you see the same behaviour using the latest released Debezium version?

Yes. Verified by inspection rather than by running, on `main` (3.7.0-SNAPSHOT), 3.6.1.Final and 3.7.0.Beta1:

- `isGtidModeEnabled()` is still the exact `"ON"` match.
- `isBinlogPositionAvailable` still opens with `if (gtid != null)` and no gtid-mode guard.

DBZ#1385 ("Allow MySQL source connector to ignore GTID on recovery", in 3.6.0+) added `gtid.ignore.on.recovery` and `prepareOffsetContextForBinlogRecovery()` -> `offsetContext.resetGtidSet()`, which is close to the right mechanism, but it is nested inside `if (isGtidModeEnabled)` within the non-GTID-recovery branch:

```java
// v3.6.1.Final BinlogStreamingChangeEventSource:280-286
else {
// The server is not using GTIDs, so start reading the binlog based upon where we last left off ...
client.setBinlogFilename(...);
client.setBinlogPosition(...);
if (isGtidModeEnabled) { // false under ON_PERMISSIVE, so the reset never runs
initializeGtidSet("");
prepareOffsetContextForBinlogRecovery(effectiveOffsetContext);
}
}
```

---

**Do you have the connector logs, ideally from start till finish?**

_You might be asked later to provide [DEBUG/TRACE](https://debezium.io/documentation/reference/stable/operations/logging.html) level log._

Yes - the relevant excerpts are quoted above. Note which log lines are *absent*: `Registering binlog reader with GTID set` and `GTID set purged on server` are emitted only inside the `if (isGtidModeEnabled)` branch of `BinlogStreamingChangeEventSource`, and neither appears once across 14 days of restarts. Every resume was by binlog file/position.

Both conditions can only hold when `gtid_mode=ON` and the operator has explicitly opted out of GTID recovery, so it does not help here.

---

**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**

1. Start MySQL with `gtid_mode=ON_PERMISSIVE` and `enforce_gtid_consistency=ON`, and a short `binlog_expire_logs_seconds` (e.g. 300).
2. Start a MySQL connector with `snapshot.mode=schema_only` and `read.only=false`. The snapshot writes the current `Executed_Gtid_Set` into the offset.
3. Generate continuous change traffic so the binlog rotates and the server purges past the GTID recorded in step 2. Confirm with `SELECT @@global.gtid_purged;`.
4. Confirm the connector is still streaming and its committed offset's `file`/`pos` are current, while its `gtids` is unchanged from step 2.
5. Restart the connector task.

Expected: the task resumes from the recorded binlog file/position. Actual: `DebeziumException: ... but this is no longer available on the server`.

---

### Implementation ideas

**1. Recognise the permissive mode.** Have `isGtidModeEnabled()` return true for `ON_PERMISSIVE` as well, since GTIDs really are being assigned. This makes GTID-based resume work, but it changes streaming behaviour: on the first restart after upgrading, an existing offset's stale GTID becomes authoritative and the connector would replay from it. It needs some thought about anonymous transactions, which `ON_PERMISSIVE` permits in the stream, and the `read.only` path, which genuinely requires `ON`.

**2. Make validation agree with the resume strategy.** Only validate the GTID when the streaming path maintains it:

```java
// BinlogConnectorConnection.isBinlogPositionAvailable
if (gtid != null && isGtidModeEnabled()) {
```

Validation then falls through to the existing binlog-file check, which is the position actually used for resume, and which still returns false when the binlog file has genuinely been purged - so no safety is lost. This also fixes the case of a server moving from `ON` to a non-`ON` mode, where the stored GTID stops being maintained.

Worth considering alongside either option: a warning at startup when the offset carries a GTID but `gtid_mode` is not `ON`, since today that state is entirely silent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.