mysql sink: long DDL timeout can execute the same DDL repeatedly

What did you do?

Run TiCDC v8.5.7 with a generic MySQL downstream and the default MySQL sink read-timeout=2m. The upstream emitted the following DDL (only the table and column names are redacted):

ALTER TABLE `<table>` MODIFY COLUMN `<column>` ENUM('<value-1>','<value-2>','<value-3>') NOT NULL;

Each client attempt waited for two minutes before timing out. The downstream operator reported that the schema eventually showed the DDL had succeeded, but TiCDC never received a successful response.

The affected DDL identifiers were:

startTs=468788157791338636
commitTs=468788158132125854

What did you expect to see?

An ambiguous client-side timeout should not cause TiCDC to blindly execute the same potentially expensive or non-idempotent DDL many times. If the DDL has already committed downstream, TiCDC should either reconcile the outcome or provide a safe way to avoid repeated downstream execution without failing the changefeed.

What did you see instead?

TiCDC received the DDL at 2026-09-01 17:48:25.593 UTC:

[2026/09/01 17:48:25.593 +00:00] [INFO] [basic_dispatcher.go:518] ["dispatcher receive ddl event"] [query="ALTER TABLE `<table>` MODIFY COLUMN `<column>` ENUM('<value-1>','<value-2>','<value-3>') NOT NULL"] [commitTs=468788158132125854]

The first execution waited for two minutes, then the MySQL driver reported a TCP read timeout. ExecContext and rollback both saw an invalid connection:

[2026/09/01 17:50:25.598 +00:00] [ERROR] [connection.go:64] ["packets.go:58 read tcp <ticdc-pod>:<port>-><mysql-endpoint>:3306: i/o timeout"] [component="[mysql]"]
[2026/09/01 17:50:25.598 +00:00] [ERROR] [mysql_writer_ddl.go:140] ["Fail to ExecContext"] [err="invalid connection"] [query="ALTER TABLE `<table>` MODIFY COLUMN `<column>` ENUM('<value-1>','<value-2>','<value-3>') NOT NULL"]
[2026/09/01 17:50:25.598 +00:00] [ERROR] [mysql_writer_ddl.go:151] ["Failed to rollback"] [sql="ALTER TABLE `<table>` MODIFY COLUMN `<column>` ENUM('<value-1>','<value-2>','<value-3>') NOT NULL"] [error="invalid connection"]
[2026/09/01 17:50:25.598 +00:00] [WARN] [mysql_writer_ddl.go:215] ["Execute DDL with error, retry later"] [ddl="ALTER TABLE `<table>` MODIFY COLUMN `<column>` ENUM('<value-1>','<value-2>','<value-3>') NOT NULL"] [startTs=468788157791338636] [commitTs=468788158132125854] [error="invalid connection"]

The same ExecContext error was logged 40 times. The first internal retry cycle exhausted 20 attempts, then the changefeed restart path caused a second 20-attempt cycle:

attempt  1: 2026-09-01 17:50:25 UTC  Fail to ExecContext, err="invalid connection"
attempt 10: 2026-09-01 18:08:50 UTC  Fail to ExecContext, err="invalid connection"
attempt 20: 2026-09-01 18:29:25 UTC  Fail to ExecContext, err="invalid connection"
attempt 21: 2026-09-01 18:34:45 UTC  Fail to ExecContext, err="invalid connection"
attempt 30: 2026-09-01 18:53:09 UTC  Fail to ExecContext, err="invalid connection"
attempt 40: 2026-09-01 19:13:40 UTC  Fail to ExecContext, err="invalid connection"

After the 40th execution attempt, the second internal retry cycle reached its limit and the changefeed entered failed:

[2026/09/01 19:13:40.447 +00:00] [ERROR] [dispatcher_manager.go:524] ["Event Dispatcher Manager Meets Error"] [error="[CDC:ErrReachMaxTry]reach maximum try: 20, error: [CDC:ErrExecDDLFailed] ... invalid connection"]
[2026/09/01 19:13:40.496 +00:00] [ERROR] [backoff.go:210] ["The changefeed won't be restarted as it has been experiencing failures for an extended duration"] [maxElapsedTime=30m0s] [checkpointTs=468788158132125853]
[2026/09/01 19:13:40.496 +00:00] [INFO] [controller.go:493] ["changefeed status changed"] [state=failed] [error="... [CDC:ErrReachMaxTry] ... [CDC:ErrExecDDLFailed] ... invalid connection"]

The checkpoint remained immediately before the DDL commit TS even though the downstream operator reported that the schema contained the requested change.

Root cause

In this version:

• The default MySQL sink read timeout is two minutes: defaultReadTimeout = "2m".
• The timeout is installed as the driver's socket readTimeout: dsn.Params["readTimeout"] = cfg.ReadTimeout.
• DDL execution retries up to 20 times. An invalid-connection outcome is reconciled through TiDB DDL job metadata only when cfg.IsTiDB is true. For a generic MySQL downstream, the code returns ErrExecDDLFailed and submits the DDL again: execDDLWithMaxRetries.

Why TiDB can reconcile the outcome but generic MySQL cannot

TiCDC detects a TiDB downstream by calling SELECT tidb_version() and records the result in cfg.IsTiDB: CheckIsTiDB and GenerateDSN.

For a TiDB downstream, TiCDC records an approximate DDL creation time using TIDB_PARSE_TSO(@@tidb_current_ts) before execution. If ExecContext later returns mysql.ErrInvalidConn, the cfg.IsTiDB branch calls waitDDLDone, reconnects, and queries information_schema.ddl_jobs using the DDL query and creation time. A done or synced job is treated as successful replication; a running or queueing job is waited on (except the existing asynchronous ADD INDEX handling), while cancelled or rollback states are treated as failures:

• getDDLCreateTime
• waitDDLDone
• getDDLStateFromTiDB

A generic MySQL-compatible downstream does not expose TiDB's TIDB_PARSE_TSO, @@tidb_current_ts, or information_schema.ddl_jobs. Therefore cfg.IsTiDB is false and the TiDB reconciliation branch is unavailable. After the client-side timeout, TiCDC cannot determine whether the server committed the DDL; it falls back to returning a retryable error and executing the same DDL again.

The timeout is therefore an ambiguous outcome: the downstream may commit the DDL after TiCDC stops waiting, while TiCDC cannot observe the success response and retries it.

Downstream side effects

• The same DDL can be submitted repeatedly or overlap with a still-running prior execution.
• A long ALTER TABLE may repeatedly rebuild or scan the table, consume CPU/disk I/O, produce binlog traffic, and hold or wait for metadata locks.
• DML and later DDLs can remain blocked behind the repeated DDL for more than an hour.
• A schema-idempotent DDL may leave the final schema unchanged but still repeat all operational costs. A non-idempotent DDL may apply an unintended mutation more than once or fail differently after the first successful execution.
• TiCDC keeps its checkpoint before the DDL and eventually fails the changefeed, even when the downstream schema already reflects the DDL.

Workaround

After verifying that the downstream DDL completed successfully and that the resulting schema exactly matches the upstream schema, configure the changefeed to ignore this specific DDL transaction by its start TS, then resume it:

[filter]
ignore-txn-start-ts = [468788157791338636]

This intentionally skips sending this DDL to the downstream. It must not be used unless the downstream schema has already been verified; otherwise subsequent DML may be replicated against an incompatible schema.

Versions of the cluster

Upstream TiDB cluster version:

v8.5.3

Upstream TiKV version:

v8.5.3

TiCDC version:

v8.5.7-20260728-c8d4d68
commit c8d4d6802d618e5800ca09e0c5e1d8eb1ff0891c

Downstream:

MySQL-compatible server (not TiDB); exact version unavailable

Start with pkg/sink/mysql/mysql_writer_ddl.go, especially execDDLWithMaxRetries and the invalid-connection handling, then read pkg/sink/mysql/helper.go for CheckIsTiDB, getDDLCreateTime, waitDDLDone, and getDDLStateFromTiDB. Trace the generic MySQL path and define a safe completion or retry behavior that avoids blindly re-executing an ambiguous DDL while preserving correct checkpoint handling.