TiCDC cannot correctly raw-replay RECOVER TABLE when TiDB recovers schema from DROP TABLE StartTS snapshot
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Problem
TiCDC currently treats RECOVER TABLE as a raw replayable DDL for MySQL/TiDB sink.
This is not deterministic when source TiDB's recovered TableInfo differs from
what downstream TiDB reconstructs from its own DDL history.
The underlying TiDB behavior is also surprising: RECOVER TABLE can recover a
schema that is not consistent with the commitTs-linear DDL history.
Concrete example
The failing table instance was:
database/table: db3.t12_r_5993561
tableID: 776
failure: Error 1054 (42S22): Unknown column 'b' in 'field list'
Source DDL commit order:
ALTER TABLE `db3`.`t12_r_5993561` DROP COLUMN `b`
commitTs/finishedTs: 467264036656382412
physical time: 2026-06-26 18:47:22 +0800
DROP TABLE IF EXISTS `db3`.`t12_r_5993561`
commitTs/finishedTs: 467264036722180751
physical time: 2026-06-26 18:47:23 +0800
RECOVER TABLE `db3`.`t12_r_5993561`
commitTs/finishedTs: 467264039500645175
physical time: 2026-06-26 18:47:33 +0800
So the commitTs order is:
DROP COLUMN `b` -> DROP TABLE -> RECOVER TABLE
From a commitTs-linear schema-history perspective, the recovered table should not
contain column b, because DROP COLUMN b committed before both the drop-table
DDL and the recover-table DDL.
However, source TiDB recovered a TableInfo that still contained column b.
Why source TiDB recovered column b
The source TiDB DDL job timing was:
18:47:22.795 job 963 submitted: ALTER TABLE ... DROP COLUMN `b`
18:47:22.948 job 963 state: delete reorganization
18:47:22.981 job 964 submitted: DROP TABLE IF EXISTS ...
job 964 start time: 2026-06-26 18:47:22.958 +0800
18:47:22.999 job 963 state: done
18:47:23.014 job 963 finished/synced
18:47:23.286 job 964 finished/synced
18:47:33.735 job 1021 submitted: RECOVER TABLE ...
18:47:33.941 job 1021 finished/synced
TiDB's RECOVER TABLE implementation finds the historical DROP TABLE job and
uses that job's StartTS to read snapshot table meta. In this case:
DROP TABLE job StartTS = 18:47:22.958
DROP COLUMN b done = 18:47:22.999
Therefore the source recover-table DDL used a snapshot from before column b
was fully dropped, even though DROP COLUMN b committed before DROP TABLE and
RECOVER TABLE.
Relevant TiDB code path:
pkg/executor/ddl.go
- executeRecoverTable obtains RecoverTableInfo.
- recoverInfo.SnapshotTS is set from the historical drop/truncate job StartTS.
- GetDropOrTruncateTableInfoFromJobs reads table meta via GetSnapshotMeta(startTS).
pkg/ddl/table.go
- onRecoverTable uses recoverInfo.TableInfo as the recovered table info.
This means the source TiDB recover result is explainable by the current
implementation, but it is not equivalent to commitTs-linear DDL schema semantics.
Why TiCDC cannot correctly raw-replay this
TiCDC sees the DDLs in the correct commitTs order:
DROP COLUMN `b` -> DROP TABLE -> RECOVER TABLE
TiCDC also receives the source-side recovered TableInfo, which contains b.
The DML SQL builder later uses that source-side TableInfo.
The problem is that MySQL/TiDB sink executes the raw query:
RECOVER TABLE `db3`.`t12_r_5993561`
on downstream TiDB. Downstream TiDB evaluates this DDL using downstream-local DDL
history and the downstream-local drop-table job StartTS, not the source
drop-table job StartTS and not the source recovered TableInfo.
In the observed run, downstream timing differed:
18:55:33.907 downstream finished ALTER TABLE ... DROP COLUMN `b`
18:58:13.317 downstream submitted DROP TABLE ...
downstream drop-table job start time: 18:58:13.296
18:58:13.409 downstream finished DROP TABLE
19:01:58.099 downstream submitted RECOVER TABLE ...
So downstream recovered the table from a snapshot after column b had already
been dropped. The downstream recovered table did not contain b.
After that, TiCDC emitted DML according to the source recovered TableInfo:
REPLACE INTO `db3`.`t12_r_5993561` (`id`,`a`,`c`,`d`,`e`,`bin`,`b`)
VALUES (?,?,?,?,?,?,?)
Downstream TiDB rejected it:
Error 1054 (42S22): Unknown column 'b' in 'field list'
TiCDC code path
Relevant TiCDC behavior:
logservice/schemastore/persist_storage_ddl_handlers.go
- PersistedDDLEvent.TableInfo is copied from job.BinlogInfo.TableInfo.
- ActionRecoverTable is handled like a new-table DDL.
logservice/schemastore/multi_version.go
- ActionRecoverTable resets deleteVersion and appends the recovered TableInfo.
pkg/sink/mysql/mysql_writer.go
- FlushDDLEvent executes DDL if downstream is TiDB or the event is not TiDBOnly.
pkg/sink/mysql/mysql_writer_ddl.go
- execDDL executes event.GetDDLQuery() directly.
pkg/sink/mysql/sql_builder.go
- DML column list is built from TableInfo.GetColumns().
The SQL builder is not inventing column b; it is faithfully using the
source-side recovered TableInfo.
Expected behavior
TiCDC should not assume that raw RECOVER TABLE replay is deterministic.
Possible directions:
- Mark/filter
ActionRecoverTableas unsupported for MySQL/TiDB sink replication. - Implement dedicated deterministic recover-table handling instead of raw replay.
The second option is more complex than generating CREATE TABLE from
TableInfo, because RECOVER TABLE also restores historical table data and
metadata. It likely needs explicit product semantics between TiDB and TiCDC.
Separately, TiDB may need to revisit whether RECOVER TABLE should recover
schema according to commitTs-linear DDL history instead of the drop-table job
StartTS snapshot.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with logservice/schemastore/persist_storage_ddl_handlers.go and multi_version.go to trace how ActionRecoverTable and its TableInfo are persisted, then follow pkg/sink/mysql/mysql_writer.go, mysql_writer_ddl.go, and sql_builder.go. Determine whether filtering or dedicated handling is required, and define completion as preventing source and downstream recovered schemas from diverging during replication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100