keyspace: in-place upgrade of an already-bootstrapped cluster does not create the system keyspace
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 33
Description
## Bug Report
### What did you do?
Upgrade an already-bootstrapped PD cluster in place to a PD version that introduces the reserved system keyspace (NextGen kernel, `constant.SystemKeyspaceID` / `SYSTEM`), without going through a fresh cluster bootstrap.
Relevant code path:
- The only place that creates the reserved bootstrap keyspace (`SYSTEM` for NextGen, `DEFAULT` for classic) is `keyspace.Manager.Bootstrap()` (`pkg/keyspace/keyspace.go:172`), via `initReserveKeyspace(GetBootstrapKeyspaceID(), GetBootstrapKeyspaceName())`.
- `Manager.Bootstrap()` is only invoked from `Server.bootstrapCluster()` (`server/server.go:843`), which itself only runs when a store sends the `Bootstrap` RPC to a cluster whose `ClusterPath` etcd key doesn't exist yet (guarded by an etcd txn with `CreateRevision(ClusterPath) == 0`, `server/server.go:819`).
- For a cluster that was already bootstrapped by an older PD binary (before the system keyspace concept existed, or before the store started running with the NextGen kernel type), `ClusterPath` already exists in etcd. On in-place upgrade/restart, PD takes the "already bootstrapped" path (`Server.createRaftCluster()` → `s.cluster.Start(s, false)`, `server/server.go:852-858`), which never calls `KeyspaceManager.Bootstrap()`.
- There is no separate, idempotent "ensure reserved keyspace exists" step that runs on normal startup for an already-bootstrapped cluster — keyspace bootstrap only happens once, at first-ever cluster bootstrap.
### What did you expect to see?
After upgrading PD in place, the reserved system keyspace should exist (created/backfilled if missing), since other components assume its presence (e.g. TSO keyspace group routing in `pkg/tso/keyspace_group_manager.go:84` and `pkg/utils/tsoutil/tso_request.go:66` resolve requests to `constant.SystemKeyspaceID`).
### What did you see instead?
An already-bootstrapped cluster that upgrades PD in place can end up with no `SYSTEM` keyspace at all, because keyspace bootstrap is gated behind the one-time cluster-bootstrap flow rather than being ensured on every startup. This can surface later as failures/inconsistent state anywhere that assumes the system keyspace exists (TSO keyspace-group resolution, GC management, etc.).
### What version of PD are you using (`pd-server -V`)?
N/A — found via code review of `pkg/keyspace/keyspace.go` and `server/server.go`'s bootstrap/startup paths; reproducible on any pre-existing cluster upgrading in place to a PD version where the system keyspace was newly introduced.
Contributor guide
Research direction
Read pkg/keyspace/keyspace.go:172 and the bootstrap/startup paths in server/server.go:819-858. Trace how an already-bootstrapped cluster starts without calling KeyspaceManager.Bootstrap(), then check the system-keyspace assumptions in pkg/tso/keyspace_group_manager.go:84 and pkg/utils/tsoutil/tso_request.go:66. Done means an in-place upgrade creates or backfills the reserved SYSTEM keyspace without requiring a fresh cluster bootstrap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100