pingcap / pingcap/ticdc

TiCDC cannot correctly raw-replay RECOVER TABLE when TiDB recovers schema from DROP TABLE StartTS snapshot

Open
#5,526 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Problem

TiCDC currently treats RECOVER TABLE as a raw replayable DDL for MySQL/TiDB sink.
This is not deterministic when source TiDB's recovered TableInfo differs from
what downstream TiDB reconstructs from its own DDL history.

The underlying TiDB behavior is also surprising: RECOVER TABLE can recover a
schema that is not consistent with the commitTs-linear DDL history.

Concrete example

The failing table instance was:

database/table: db3.t12_r_5993561
tableID:        776
failure:        Error 1054 (42S22): Unknown column 'b' in 'field list'

Source DDL commit order:

ALTER TABLE `db3`.`t12_r_5993561` DROP COLUMN `b`
  commitTs/finishedTs: 467264036656382412
  physical time:       2026-06-26 18:47:22 +0800

DROP TABLE IF EXISTS `db3`.`t12_r_5993561`
  commitTs/finishedTs: 467264036722180751
  physical time:       2026-06-26 18:47:23 +0800

RECOVER TABLE `db3`.`t12_r_5993561`
  commitTs/finishedTs: 467264039500645175
  physical time:       2026-06-26 18:47:33 +0800

So the commitTs order is:

DROP COLUMN `b` -> DROP TABLE -> RECOVER TABLE

From a commitTs-linear schema-history perspective, the recovered table should not
contain column b, because DROP COLUMN b committed before both the drop-table
DDL and the recover-table DDL.

However, source TiDB recovered a TableInfo that still contained column b.

Why source TiDB recovered column b

The source TiDB DDL job timing was:

18:47:22.795  job 963 submitted: ALTER TABLE ... DROP COLUMN `b`
18:47:22.948  job 963 state: delete reorganization
18:47:22.981  job 964 submitted: DROP TABLE IF EXISTS ...
               job 964 start time: 2026-06-26 18:47:22.958 +0800
18:47:22.999  job 963 state: done
18:47:23.014  job 963 finished/synced
18:47:23.286  job 964 finished/synced
18:47:33.735  job 1021 submitted: RECOVER TABLE ...
18:47:33.941  job 1021 finished/synced

TiDB's RECOVER TABLE implementation finds the historical DROP TABLE job and
uses that job's StartTS to read snapshot table meta. In this case:

DROP TABLE job StartTS = 18:47:22.958
DROP COLUMN b done     = 18:47:22.999

Therefore the source recover-table DDL used a snapshot from before column b
was fully dropped, even though DROP COLUMN b committed before DROP TABLE and
RECOVER TABLE.

Relevant TiDB code path:

pkg/executor/ddl.go
- executeRecoverTable obtains RecoverTableInfo.
- recoverInfo.SnapshotTS is set from the historical drop/truncate job StartTS.
- GetDropOrTruncateTableInfoFromJobs reads table meta via GetSnapshotMeta(startTS).

pkg/ddl/table.go
- onRecoverTable uses recoverInfo.TableInfo as the recovered table info.

This means the source TiDB recover result is explainable by the current
implementation, but it is not equivalent to commitTs-linear DDL schema semantics.

Why TiCDC cannot correctly raw-replay this

TiCDC sees the DDLs in the correct commitTs order:

DROP COLUMN `b` -> DROP TABLE -> RECOVER TABLE

TiCDC also receives the source-side recovered TableInfo, which contains b.
The DML SQL builder later uses that source-side TableInfo.

The problem is that MySQL/TiDB sink executes the raw query:

RECOVER TABLE `db3`.`t12_r_5993561`

on downstream TiDB. Downstream TiDB evaluates this DDL using downstream-local DDL
history and the downstream-local drop-table job StartTS, not the source
drop-table job StartTS and not the source recovered TableInfo.

In the observed run, downstream timing differed:

18:55:33.907  downstream finished ALTER TABLE ... DROP COLUMN `b`
18:58:13.317  downstream submitted DROP TABLE ...
               downstream drop-table job start time: 18:58:13.296
18:58:13.409  downstream finished DROP TABLE
19:01:58.099  downstream submitted RECOVER TABLE ...

So downstream recovered the table from a snapshot after column b had already
been dropped. The downstream recovered table did not contain b.

After that, TiCDC emitted DML according to the source recovered TableInfo:

REPLACE INTO `db3`.`t12_r_5993561` (`id`,`a`,`c`,`d`,`e`,`bin`,`b`)
VALUES (?,?,?,?,?,?,?)

Downstream TiDB rejected it:

Error 1054 (42S22): Unknown column 'b' in 'field list'

TiCDC code path

Relevant TiCDC behavior:

logservice/schemastore/persist_storage_ddl_handlers.go
- PersistedDDLEvent.TableInfo is copied from job.BinlogInfo.TableInfo.
- ActionRecoverTable is handled like a new-table DDL.

logservice/schemastore/multi_version.go
- ActionRecoverTable resets deleteVersion and appends the recovered TableInfo.

pkg/sink/mysql/mysql_writer.go
- FlushDDLEvent executes DDL if downstream is TiDB or the event is not TiDBOnly.

pkg/sink/mysql/mysql_writer_ddl.go
- execDDL executes event.GetDDLQuery() directly.

pkg/sink/mysql/sql_builder.go
- DML column list is built from TableInfo.GetColumns().

The SQL builder is not inventing column b; it is faithfully using the
source-side recovered TableInfo.

Expected behavior

TiCDC should not assume that raw RECOVER TABLE replay is deterministic.

Possible directions:

  1. Mark/filter ActionRecoverTable as unsupported for MySQL/TiDB sink replication.
  2. Implement dedicated deterministic recover-table handling instead of raw replay.

The second option is more complex than generating CREATE TABLE from
TableInfo, because RECOVER TABLE also restores historical table data and
metadata. It likely needs explicit product semantics between TiDB and TiCDC.

Separately, TiDB may need to revisit whether RECOVER TABLE should recover
schema according to commitTs-linear DDL history instead of the drop-table job
StartTS snapshot.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with logservice/schemastore/persist_storage_ddl_handlers.go and multi_version.go to trace how ActionRecoverTable and its TableInfo are persisted, then follow pkg/sink/mysql/mysql_writer.go, mysql_writer_ddl.go, and sql_builder.go. Determine whether filtering or dedicated handling is required, and define completion as preventing source and downstream recovered schemas from diverging during replication.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.