TiCDC can silently replicate wrong values for online MODIFY COLUMN when DML commitTs precedes DDL finishedTs

Bug Report

TiCDC can silently replicate wrong values when TiDB online MODIFY COLUMN runs concurrently with DML and the DML commitTs is earlier than the DDL finishedTs, but the final upstream row value already depends on the new column schema.

The changefeed reached the final sync_diff, and sync_diff found data inconsistency while table structure and row counts were equal.

This is also not new-arch-only. The old arch TiCDC/TiFlow mounter has the same class of risk because it decodes DML with the snapshot at raw.CRTs - 1. If DML commitTs < MODIFY COLUMN finishedTs, old arch also decodes the DML with the old schema, writes it before the DDL, and executes the DDL only after the checkpoint reaches the DDL commitTs.

Test Information

Remote test machine:

branch: codex/weekly-rand-ddl-debug-20260628
TiCDC commit: 00e40aad2b73677d606189e93660c5fe37bad51a
case: weekly_rand_multi
run dir: /tmp/tidb_cdc_test/codex-rerun-weekly-rand-multi-20260630-100507
case dir: /tmp/tidb_cdc_test/weekly_rand_multi

Command:

PATH=/usr/local/go/bin:$PATH
GOTOOLCHAIN=auto
TICDC_NEWARCH=true
RUN_PROFILE=weekly
RUN_DURATION=30m
RUN_SEED=2026063001
timeout 120m tests/integration_tests/run.sh mysql weekly_rand_multi

Actual Result

The final sync_diff failed. Only one table was inconsistent:

[2026/06/30 11:31:07.448 +08:00] sync_diff started checking the final state
exit_status=1
end_time=2026-06-30T11:31:39+08:00

TABLE                   RESULT   STRUCTURE EQUALITY   DATA DIFF ROWS   UPCOUNT   DOWNCOUNT
`db5`.`t12_r_1313262`   succeed  true                 +3/-3            1453      1453

The inconsistent rows were:

-- source c='88.5', target c='88.50'
REPLACE INTO `db5`.`t12_r_1313262`(`id`,`a`,`b`,`c`,`d`)
VALUES (12223,'331529','C9tBByfl7r50OgLcT8E9YeIPyw4WhA2v','88.5','2023-11-15 01:37:03');

-- source c='18.55', target c='0.00'
REPLACE INTO `db5`.`t12_r_1313262`(`id`,`a`,`b`,`c`,`d`)
VALUES (12224,'47643','3qNtzigXTgBJZcAmUWcpDpFpws83NTzZ','18.55','2023-11-15 01:37:04');

-- source a='317262', target a='0'
REPLACE INTO `db5`.`t12_r_1313262`(`id`,`a`,`b`,`c`,`d`)
VALUES (13445,'317262','K12OfroM4tkZmLM9HYDtg366ciwpbgPy','ujQltwNZxp12mmAHrJ8zXBYlig7tWhps','2023-11-15 01:57:25');

Concrete Example 1: c was changed from DECIMAL(10,2) to VARCHAR(64)

The target physical table ID was 4705.

Before the DDL, TiCDC received and wrote DMLs for rows 12223 and 12224:

[2026/06/30 10:39:01.402 +08:00] [DEBUG] [basic_dispatcher.go:606] ["dispatcher receive all event"] [eventType=DMLEvent] [event="DMLEvent{... PhysicalTableID: 4705, StartTs: 467346820980801556, CommitTs: 467346820980801580, Table: db5.t12_r_1313262, ... Rows: Insert: Row: 12223, 331529, C9tBByfl7r50OgLcT8E9YeIPyw4WhA2v, 88.50, 2023-11-15 01:37:03, ...;}"]
[2026/06/30 10:39:01.403 +08:00] [DEBUG] [sql_builder.go:101] ["... Query: REPLACE INTO `db5`.`t12_r_1313262` (`id`,`a`,`b`,`c`,`d`,`e`) VALUES (?,?,?,?,?,?), Args: (12223, 331529, C9tBByfl7r50OgLcT8E9YeIPyw4WhA2v, 88.50, 2023-11-15 01:37:03, ...),CommitTs: [467346820980801580],StartTs:  [467346820980801556],End"]

[2026/06/30 10:39:01.418 +08:00] [DEBUG] [basic_dispatcher.go:606] ["dispatcher receive all event"] [eventType=DMLEvent] [event="DMLEvent{... PhysicalTableID: 4705, StartTs: 467346820993646673, CommitTs: 467346820993646678, Table: db5.t12_r_1313262, ... Rows: Insert: Row: 12224, 47643, 3qNtzigXTgBJZcAmUWcpDpFpws83NTzZ, 0, 2023-11-15 01:37:04, ...;}"]
[2026/06/30 10:39:01.418 +08:00] [DEBUG] [sql_builder.go:101] ["... Query: REPLACE INTO `db5`.`t12_r_1313262` (`id`,`a`,`b`,`c`,`d`,`e`) VALUES (?,?,?,?,?,?), Args: (12224, 47643, 3qNtzigXTgBJZcAmUWcpDpFpws83NTzZ, 0, 2023-11-15 01:37:04, ...),CommitTs: [467346820993646678],StartTs:  [467346820993646673],End"]

Then TiCDC received and executed the DDL:

[2026/06/30 10:39:01.417 +08:00] [INFO] [event_broker.go:254] ["send ddl event to dispatcher"] [query="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `c` VARCHAR(64) NOT NULL"] [commitTs=467346821006753986] [seq=51]
[2026/06/30 10:39:01.421 +08:00] [DEBUG] [basic_dispatcher.go:606] ["dispatcher receive all event"] [eventType=DDLEvent] [event="DDLEvent{... Query: ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `c` VARCHAR(64) NOT NULL, ... StartTs: 467346821006753912, FinishedTs: 467346821006753986, Seq: 51, ...}"]
[2026/06/30 10:39:01.597 +08:00] [INFO] [mysql_writer_ddl.go:242] ["Execute DDL succeeded"] [changefeed=default/weeklyrand] [startTs=467346821006753912] [commitTs=467346821006753986] [query="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `c` VARCHAR(64) NOT NULL"]

The ordering seen by TiCDC was:

row 12223 DML commitTs = 467346820980801580
row 12224 DML commitTs = 467346820993646678
DDL finishedTs         = 467346821006753986

467346820980801580 < 467346820993646678 < 467346821006753986

So TiCDC had to write both DMLs before executing the c VARCHAR(64) DDL. However, the final upstream value after the DDL was c='88.5' / c='18.55', while TiCDC had already decoded/written the earlier DML as 88.50 and 0.

Concrete Example 2: a was changed from INT to VARCHAR(64)

For row id=13445, TiCDC had already decoded the DML value as a=0 before the a VARCHAR(64) DDL:

[2026/06/30 10:43:12.459 +08:00] [DEBUG] [basic_dispatcher.go:606] ["dispatcher receive all event"] [eventType=DMLEvent] [event="DMLEvent{... PhysicalTableID: 4705, StartTs: 467346910791073913, CommitTs: 467346910791073964, Table: db5.t12_r_1313262, ... Rows: Insert: Row: 13445, 0, K12OfroM4tkZmLM9HYDtg366ciwpbgPy, ujQltwNZxp12mmAHrJ8zXBYlig7tWhps, 2023-11-15 01:57:25;}"]
[2026/06/30 10:43:12.467 +08:00] [DEBUG] [sql_builder.go:101] ["... Query: REPLACE INTO `db5`.`t12_r_1313262` (`id`,`a`,`b`,`c`,`d`) VALUES (?,?,?,?,?), Args: (13445, 0, K12OfroM4tkZmLM9HYDtg366ciwpbgPy, ujQltwNZxp12mmAHrJ8zXBYlig7tWhps, 2023-11-15 01:57:25),CommitTs: [467346910791073964],StartTs:  [467346910791073913],End"]

Then the DDL was executed:

[2026/06/30 10:43:17.956 +08:00] [INFO] [mysql_writer_ddl.go:242] ["Execute DDL succeeded"] [changefeed=default/weeklyrand] [startTs=467346910791073963] [commitTs=467346910804181101] [query="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `a` VARCHAR(64) NOT NULL"]

The ordering was:

row 13445 DML commitTs = 467346910791073964
DDL finishedTs         = 467346910804181101

467346910791073964 < 467346910804181101

The final upstream row had a='317262', but TiCDC had decoded and written a=0. This proves the value was already wrong in the CDC decoding path, not merely converted by downstream SQL execution.

Relevant upstream DDL trace

The runner trace uses UTC, while TiDB/TiCDC logs above use +08:00. The same target table had successful MODIFY COLUMN DDLs in the workload:

2026/06/30 02:36:15.499145 kind=modify_column_type target=`db5`.`t12_r_1313262` status=ok sql="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `c` VARCHAR(64) NOT NULL" err=""
2026/06/30 02:36:23.112955 kind=modify_column_type target=`db5`.`t12_r_1313262` status=ok sql="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `a` VARCHAR(64) NOT NULL" err=""

SchemaStore also persisted those DDL events with the same finishedTs:

[2026/06/30 10:36:15.068 +08:00] [DEBUG] [schema_store.go:173] ["write ddl event"] [schemaID=124] [tableID=4705] [finishedTs=467346908785934415] [query="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `c` VARCHAR(64) NOT NULL"]
[2026/06/30 10:36:22.772 +08:00] [DEBUG] [schema_store.go:173] ["write ddl event"] [schemaID=124] [tableID=4705] [finishedTs=467346910804181101] [query="ALTER TABLE `db5`.`t12_r_1313262` MODIFY COLUMN `a` VARCHAR(64) NOT NULL"]

Why old arch TiCDC/TiFlow has the same class of risk

Old arch investigation was done against pingcap/tiflow at:

commit: 979ae7086fbfe9e0f6b4d1a69aa23fb4e1cfcd11

The old arch mounter decodes DML using the schema snapshot at raw.CRTs - 1:

// cdc/entry/mounter.go
// A DML whose commitTs is equal to a DDL finishedTs should use the schema info before the DDL.
snap, err := m.schemaStorage.GetSnapshot(ctx, raw.CRTs-1)

Schema storage creates new snapshots at DDL finishedTs, so:

DML commitTs < DDL finishedTs  => old arch mounter uses the old schema
DML commitTs == DDL finishedTs => old arch mounter also uses the old schema because of raw.CRTs - 1
DML commitTs > DDL finishedTs  => old arch mounter can use the new schema

Old arch then writes all DMLs whose commitTs are below the pending DDL barrier before executing the DDL. So for this same TiDB history:

DML commitTs < MODIFY COLUMN finishedTs

old arch would also decode the DML with the old schema and write it before the downstream MODIFY COLUMN.

If TiDB online DDL has already made that DML's final value depend on the new schema or a new column ID, old arch can produce the same silent wrong values:

• DECIMAL(10,2) -> VARCHAR(64): upstream final value can be '88.5', while old-schema decoding/writing preserves '88.50'.
• Missing old column ID during the transition can make old arch call getDefaultOrZeroValue, producing values such as 0, 0.00, empty string, or NULL.
• INT -> VARCHAR(64): upstream final value can be '317262', while old-schema/missing-column decoding can write 0, which remains wrong after downstream DDL.

Therefore old arch is not a safe counterexample. It may only be correct accidentally when TiDB still maintains the old column ID and the old type can represent the value losslessly.

Root Cause Hypothesis

TiCDC's replication model assumes this ordering contract:

DML commitTs < DDL finishedTs  => decode and write DML with the old schema before the DDL
DML commitTs > DDL finishedTs  => decode and write DML with the new schema after the DDL

TiDB online MODIFY COLUMN can violate this assumption. During online DDL state transitions and reorg, DML is not blocked for the full DDL lifetime. A row's final upstream value may already depend on the new schema while the DML commitTs is still smaller than the DDL job finishedTs.

In that case, CDC only sees:

DML commitTs < DDL finishedTs

and cannot infer that the DML should be decoded with the new schema or replayed after an earlier schema barrier.

Expected Behavior

TiCDC should not silently produce inconsistent downstream data for online MODIFY COLUMN with concurrent DML.

Possible directions:

1. TiDB provides a schema barrier / schema version timestamp that represents when DML starts using the new schema semantics, not only the final DDL finishedTs.
2. TiCDC obtains the real schema version or column mapping for each DML event.
3. TiCDC detects this unsupported history and fails loudly instead of silently writing wrong values.
4. The limitation is documented and random DDL tests avoid unsafe MODIFY COLUMN patterns until the contract is fixed.

Actual Behavior

TiCDC new arch completed the replication, but the final downstream data differed:

id=12223: source c='88.5',   target c='88.50'
id=12224: source c='18.55',  target c='0.00'
id=13445: source a='317262', target a='0'

Old arch TiCDC/TiFlow has the same class of risk because it decodes DML commitTs < DDL finishedTs with old schema and executes DML before the DDL.