schemastore: lazy keyspace initialization can block graceful shutdown
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
What did you do?
Run the NextGen storage-sink fail_over integration test with two TiCDC
nodes. The test creates a changefeed for a non-default keyspace, terminates one
node, starts its replacement, and then terminates the other node while the new
coordinator is bootstrapping the changefeed.
The failure was observed in
pull-cdc-storage-integration-heavy-next-gen/723,
storage group G13.
What did you expect to see?
SIGTERM should cancel any in-flight lazy schema-store initialization. The TiCDC
process should close within the graceful-shutdown deadline, allowing failover
or scale-in automation to start a replacement safely.
What did you see instead?
The process accepted SIGTERM but did not exit within 30 seconds:
[18:49:29.313] got signal, prepare to shutdown
[18:49:29.316] schema store initialize from kv storage begin
[18:49:29-18:49:58] loadRegion from PD failed ... ErrClientGetProtoClient
[18:49:59.315] graceful shutdown timeout, exit server
wait process 4983 exit timeout
The shutdown race is:
getKeyspaceSchemaStorestarts lazy registration using
context.Background().RegisterKeyspaceholdskeyspaceLockerwhile constructing the persistent
schema store.persistSchemaSnapshotWithEncryptionretriesListDatabasesindefinitely
and does not observe a cancellation context.- During shutdown, the PD client is closed and the snapshot initialization
keeps retrying.SchemaStore.Closewaits forkeyspaceLocker, so the server
reaches its graceful-shutdown timeout.
This can make graceful scale-in or failover report an error and leave the old
process alive until an external hard-kill deadline. No data inconsistency was
observed in this run; the test stopped before Sync Diff.
Suggested direction:
- propagate a server-lifecycle context into lazy keyspace registration;
- make schema snapshot retry loops cancellation-aware; and
- avoid holding the global keyspace lock across unbounded I/O, or otherwise
allowCloseto cancel and wait for in-flight initialization.
Versions of the cluster
Upstream TiDB cluster version:
CLOUD.202609.0-10f06594e7
Upstream TiKV version:
Not captured
TiCDC version:
v8.5.4-nextgen.202510.5-417-g66509e02
Presubmit merge: base 6c03227f9 + head 339b9b66a
Kernel: NextGen
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at getKeyspaceSchemaStore and RegisterKeyspace, then trace persistSchemaSnapshotWithEncryption and SchemaStore.Close. Reproduce the two-node fail_over integration scenario and follow shutdown and retry behavior around the keyspace lock. Done means SIGTERM cancels lazy schema-store initialization and the process exits within the graceful-shutdown deadline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100