storage-consumer can reorder rename DDL and DML across scan rounds

What did you do?

A storage integration heavy CI job failed in ddl_with_random_move_table:

• CI artifact: https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_storage_integration_heavy/670/artifact/log-G12.tar.gz

I downloaded the artifact and inspected the storage files, cdc_storage_consumer.log, cdc0.log, cdc1.log, and sync_diff.log.

The failure is not caused by TiCDC producer ordering. The producer side already drained old DML before writing the later DDL. The problem is in cmd/storage-consumer.

Detailed analysis:

1. The storage artifact contains a rename chain with DML in the middle.

   • test/table_108/meta/schema_464822157269008430_1306682657.json.log
     • query: RENAME TABLE test.table_8 TO test.table_108
     • table version: 464822157269008430
   • test/table_108/464822157269008430/2026-03-10/CDC00000000000000000001.json.log
     • contains DML on test.table_108
     • one example row is DELETE FROM test.table_108 WHERE id = 124
     • commit ts: 464822157426556984
   • test/table_8/meta/schema_464822157439664147_3139277867.json.log
     • query: RENAME TABLE test.table_108 TO test.table_8
     • table version: 464822157439664147

   So the only correct apply order is:

   1. table_8 -> table_108
   2. apply the DML written under table_108
   3. table_108 -> table_8

2. Producer-side logs show the sink ordering is already correct.

   • cdc0.log shows the sink drains old DML before the later rename-back DDL.
   • The intermediate table_108 DML file is persisted before the second rename schema file is written.

   This means producer ordering is not the bug here.

3. storage consumer applies files too early, based only on what one scan round happens to see.
   In cdc_storage_consumer.log:

   • round 268 scans only 3 keys and already sees the two rename schema files.
   • in the same round it executes:
     • RENAME TABLE test.table_8 TO test.table_108
     • RENAME TABLE test.table_108 TO test.table_8
   • only in round 269 does it discover the older DML file under table_108
   • it then tries to execute DELETE FROM test.table_108 WHERE id = 124
   • MySQL returns ERROR 1146 (42S02): Table 'test.table_108' doesn't exist

4. After that error, the consumer exits before finishing the remaining rename chain.
   sync_diff then reports downstream-only table test.table_106, which is just a later consequence of the earlier consumer failure.

Root cause:

cmd/storage-consumer currently assumes that "files visible in this scan round" are safe to apply immediately. That assumption is wrong for cloud storage.

The producer publishes a stable visibility watermark in the storage metadata file (checkpoint-ts), but the consumer does not read it at all. As a result, the consumer can:

• see a later rename DDL in one scan round,
• apply it immediately,
• and only discover an older DML file from the previous table name in the next scan round.

That reorders apply across scan rounds even though the underlying storage contents are correct.

There is also a second issue in the current implementation: it only tracks the latest files it has seen, not the files it has actually handled. So if a file is discovered before it is safe to apply, the current code has no durable pending state to retry it in a later round.

What did you expect to see?

ddl_with_random_move_table should finish successfully. The storage consumer should not apply rename DDLs before the stable checkpoint guarantees that all earlier DML files for that table version are visible.

What did you see instead?

The storage consumer reordered apply across scan rounds, then failed with a missing-table error on a DML that should have been applied before the rename-back DDL.

This eventually caused sync_diff to fail because downstream state diverged from upstream.

Versions of the cluster

Upstream TiDB cluster version (from cdc_storage_consumer.log):

Release Version: v9.0.0-beta.2.pre-1334-g5766c79
Git Commit Hash: 5766c79bbff7d2ac273d7cc7cfe71d29fbfc5488
Git Branch: HEAD
UTC Build Time: 2026-03-10 13:02:06
GoVersion: go1.25.6
Store: tikv

Upstream TiKV version:

Not available in the archived logs.

TiCDC version (from cdc1.log / cdc_storage_consumer_stdout.log):

TiCDC: v8.5.4-nextgen.202510.5-125-g482ab8b0
Git Commit Hash: 482ab8b093e902702a5c3e1506a1619fa5dbaf5e

storage consumer: v8.5.4-nextgen.202510.5-125-g9571ea96
Git Commit Hash: 9571ea96fd80b131af8992c76eadbc2e14e58f2e

Suggested fix

Fix cmd/storage-consumer instead of changing the producer path:

1. Read the storage metadata file every round and use checkpoint-ts as the stable apply watermark.
2. Defer any schema/DML key whose tableVersion is greater than the stable checkpoint.
3. Track seen files separately from handled files.
   • discovered-but-deferred keys must remain pending across later scan rounds,
   • otherwise the consumer can permanently lose a key that was seen too early.
4. Keep the existing TableVersion ordering inside a round, but only apply keys that are already below the stable checkpoint.

With that change, the consumer waits until the rename DDLs and the intermediate DML are all stably visible, and the apply order becomes:

1. rename to table_108
2. apply DML on table_108
3. rename back to table_8

Start in cmd/storage-consumer by tracing its scan-round handling, metadata reads, and seen-versus-handled file tracking. Use the ddl_with_random_move_table failure and the cited storage-consumer logs to verify watermark and TableVersion ordering. Done means deferred keys persist across rounds, files apply only below checkpoint-ts, and the rename/DML/rename sequence completes without the missing-table error.