pingcap / pingcap/ticdc

schemastore: panic after DROP, FLASHBACK, and DROP DATABASE

Open
#6,095 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-26.3 affects-8.5 severity/major type/bug
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

What did you do?

Run the following DDL sequence on the upstream TiDB cluster:

CREATE DATABASE test_db;
DROP DATABASE test_db;
FLASHBACK DATABASE test_db;
DROP DATABASE test_db;

The issue was observed on a five-node next-generation TiCDC deployment. All CDC nodes consumed the same DDL sequence and crashed when processing the second DROP DATABASE.

What did you expect to see?

TiCDC should not crash. SchemaStore should either maintain the recovered schema metadata for ActionRecoverSchema or safely handle the subsequent ActionDropSchema while preserving correct DDL history.

What did you see instead?

All CDC nodes panicked with the same stack:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10]

github.com/pingcap/ticdc/logservice/schemastore.updateDDLHistoryForSchemaDDL
github.com/pingcap/ticdc/logservice/schemastore/persist_storage_ddl_handlers.go:1068
github.com/pingcap/ticdc/logservice/schemastore.(*persistentStorage).handleDDLJob
github.com/pingcap/ticdc/logservice/schemastore/persist_storage.go:743
github.com/pingcap/ticdc/logservice/schemastore.(*keyspaceSchemaStore).tryUpdateResolvedTs
github.com/pingcap/ticdc/logservice/schemastore/schema_store.go:155

The crash repeats after each process restart because SchemaStore replays the same DDL sequence and encounters the same panic.

Root cause analysis

The first DROP DATABASE removes the schema from databaseMap. ActionRecoverSchema, generated by FLASHBACK DATABASE, is explicitly ignored by shouldSkipDDL, so the schema metadata is not restored. The second DROP DATABASE then calls updateDDLHistoryForSchemaDDL, which dereferences the missing entry without checking it:

for tableID := range args.databaseMap[args.ddlEvent.SchemaID].Tables {

This behavior is also present on the current master branch as of commit 3f0a68abf.

Related tracking issue: #753 marks ActionRecoverSchema as ignored, but ignoring it is unsafe when later schema DDL depends on the recovered metadata.

Suggested regression test

Add a SchemaStore test covering:

CreateSchema -> DropSchema -> RecoverSchema -> DropSchema

The test should verify that SchemaStore does not panic, advances its resolved position, and retains consistent database/table metadata and DDL history.

Versions of the cluster

Upstream TiDB cluster version:

Not collected

Upstream TiKV version:

Not collected

TiCDC version:

Release version: v8.5.4-nextgen.202510.8
Git hash: 8ba50489f00e01097187acb0f071811593512267

The same unsafe code path was verified in v26.3.1, v26.3.2, v26.3.3, v26.3.4, release-nextgen-202603, release-8.5, and master.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in logservice/schemastore/persist_storage_ddl_handlers.go around updateDDLHistoryForSchemaDDL, then trace handleDDLJob in persist_storage.go and resolved-position handling in schema_store.go. Add a SchemaStore regression test for CreateSchema → DropSchema → RecoverSchema → DropSchema, and verify it does not panic, advances the resolved position, and preserves consistent metadata and DDL history.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.