pingcap / pingcap/ticdc

schemastore: lazy keyspace initialization can block graceful shutdown

Open
#6,199 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

severity/moderate type/bug
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

What did you do?

Run the NextGen storage-sink fail_over integration test with two TiCDC
nodes. The test creates a changefeed for a non-default keyspace, terminates one
node, starts its replacement, and then terminates the other node while the new
coordinator is bootstrapping the changefeed.

The failure was observed in
pull-cdc-storage-integration-heavy-next-gen/723,
storage group G13.

What did you expect to see?

SIGTERM should cancel any in-flight lazy schema-store initialization. The TiCDC
process should close within the graceful-shutdown deadline, allowing failover
or scale-in automation to start a replacement safely.

What did you see instead?

The process accepted SIGTERM but did not exit within 30 seconds:

[18:49:29.313] got signal, prepare to shutdown
[18:49:29.316] schema store initialize from kv storage begin
[18:49:29-18:49:58] loadRegion from PD failed ... ErrClientGetProtoClient
[18:49:59.315] graceful shutdown timeout, exit server
wait process 4983 exit timeout

The shutdown race is:

  1. getKeyspaceSchemaStore starts lazy registration using
    context.Background().
  2. RegisterKeyspace holds keyspaceLocker while constructing the persistent
    schema store.
  3. persistSchemaSnapshotWithEncryption retries ListDatabases indefinitely
    and does not observe a cancellation context.
  4. During shutdown, the PD client is closed and the snapshot initialization
    keeps retrying. SchemaStore.Close waits for keyspaceLocker, so the server
    reaches its graceful-shutdown timeout.

This can make graceful scale-in or failover report an error and leave the old
process alive until an external hard-kill deadline. No data inconsistency was
observed in this run; the test stopped before Sync Diff.

Suggested direction:

  • propagate a server-lifecycle context into lazy keyspace registration;
  • make schema snapshot retry loops cancellation-aware; and
  • avoid holding the global keyspace lock across unbounded I/O, or otherwise
    allow Close to cancel and wait for in-flight initialization.

Versions of the cluster

Upstream TiDB cluster version:

CLOUD.202609.0-10f06594e7

Upstream TiKV version:

Not captured

TiCDC version:

v8.5.4-nextgen.202510.5-417-g66509e02
Presubmit merge: base 6c03227f9 + head 339b9b66a
Kernel: NextGen

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at getKeyspaceSchemaStore and RegisterKeyspace, then trace persistSchemaSnapshotWithEncryption and SchemaStore.Close. Reproduce the two-node fail_over integration scenario and follow shutdown and retry behavior around the keyspace lock. Done means SIGTERM cancels lazy schema-store initialization and the process exits within the graceful-shutdown deadline.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.