schemastore: panic after DROP, FLASHBACK, and DROP DATABASE
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
What did you do?
Run the following DDL sequence on the upstream TiDB cluster:
CREATE DATABASE test_db;
DROP DATABASE test_db;
FLASHBACK DATABASE test_db;
DROP DATABASE test_db;
The issue was observed on a five-node next-generation TiCDC deployment. All CDC nodes consumed the same DDL sequence and crashed when processing the second DROP DATABASE.
What did you expect to see?
TiCDC should not crash. SchemaStore should either maintain the recovered schema metadata for ActionRecoverSchema or safely handle the subsequent ActionDropSchema while preserving correct DDL history.
What did you see instead?
All CDC nodes panicked with the same stack:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10]
github.com/pingcap/ticdc/logservice/schemastore.updateDDLHistoryForSchemaDDL
github.com/pingcap/ticdc/logservice/schemastore/persist_storage_ddl_handlers.go:1068
github.com/pingcap/ticdc/logservice/schemastore.(*persistentStorage).handleDDLJob
github.com/pingcap/ticdc/logservice/schemastore/persist_storage.go:743
github.com/pingcap/ticdc/logservice/schemastore.(*keyspaceSchemaStore).tryUpdateResolvedTs
github.com/pingcap/ticdc/logservice/schemastore/schema_store.go:155
The crash repeats after each process restart because SchemaStore replays the same DDL sequence and encounters the same panic.
Root cause analysis
The first DROP DATABASE removes the schema from databaseMap. ActionRecoverSchema, generated by FLASHBACK DATABASE, is explicitly ignored by shouldSkipDDL, so the schema metadata is not restored. The second DROP DATABASE then calls updateDDLHistoryForSchemaDDL, which dereferences the missing entry without checking it:
for tableID := range args.databaseMap[args.ddlEvent.SchemaID].Tables {
This behavior is also present on the current master branch as of commit 3f0a68abf.
Related tracking issue: #753 marks ActionRecoverSchema as ignored, but ignoring it is unsafe when later schema DDL depends on the recovered metadata.
Suggested regression test
Add a SchemaStore test covering:
CreateSchema -> DropSchema -> RecoverSchema -> DropSchema
The test should verify that SchemaStore does not panic, advances its resolved position, and retains consistent database/table metadata and DDL history.
Versions of the cluster
Upstream TiDB cluster version:
Not collected
Upstream TiKV version:
Not collected
TiCDC version:
Release version: v8.5.4-nextgen.202510.8
Git hash: 8ba50489f00e01097187acb0f071811593512267
The same unsafe code path was verified in v26.3.1, v26.3.2, v26.3.3, v26.3.4, release-nextgen-202603, release-8.5, and master.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in logservice/schemastore/persist_storage_ddl_handlers.go around updateDDLHistoryForSchemaDDL, then trace handleDDLJob in persist_storage.go and resolved-position handling in schema_store.go. Add a SchemaStore regression test for CreateSchema → DropSchema → RecoverSchema → DropSchema, and verify it does not panic, advances the resolved position, and preserves consistent metadata and DDL history.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100