tikv / tikv/pd

keyspace: in-place upgrade of an already-bootstrapped cluster does not create the system keyspace

Open
#11,139 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
4d 15h
Merged PRs (30d)
33

Description

## Bug Report

### What did you do?

Upgrade an already-bootstrapped PD cluster in place to a PD version that introduces the reserved system keyspace (NextGen kernel, `constant.SystemKeyspaceID` / `SYSTEM`), without going through a fresh cluster bootstrap.

Relevant code path:
- The only place that creates the reserved bootstrap keyspace (`SYSTEM` for NextGen, `DEFAULT` for classic) is `keyspace.Manager.Bootstrap()` (`pkg/keyspace/keyspace.go:172`), via `initReserveKeyspace(GetBootstrapKeyspaceID(), GetBootstrapKeyspaceName())`.
- `Manager.Bootstrap()` is only invoked from `Server.bootstrapCluster()` (`server/server.go:843`), which itself only runs when a store sends the `Bootstrap` RPC to a cluster whose `ClusterPath` etcd key doesn't exist yet (guarded by an etcd txn with `CreateRevision(ClusterPath) == 0`, `server/server.go:819`).
- For a cluster that was already bootstrapped by an older PD binary (before the system keyspace concept existed, or before the store started running with the NextGen kernel type), `ClusterPath` already exists in etcd. On in-place upgrade/restart, PD takes the "already bootstrapped" path (`Server.createRaftCluster()` → `s.cluster.Start(s, false)`, `server/server.go:852-858`), which never calls `KeyspaceManager.Bootstrap()`.
- There is no separate, idempotent "ensure reserved keyspace exists" step that runs on normal startup for an already-bootstrapped cluster — keyspace bootstrap only happens once, at first-ever cluster bootstrap.

### What did you expect to see?

After upgrading PD in place, the reserved system keyspace should exist (created/backfilled if missing), since other components assume its presence (e.g. TSO keyspace group routing in `pkg/tso/keyspace_group_manager.go:84` and `pkg/utils/tsoutil/tso_request.go:66` resolve requests to `constant.SystemKeyspaceID`).

### What did you see instead?

An already-bootstrapped cluster that upgrades PD in place can end up with no `SYSTEM` keyspace at all, because keyspace bootstrap is gated behind the one-time cluster-bootstrap flow rather than being ensured on every startup. This can surface later as failures/inconsistent state anywhere that assumes the system keyspace exists (TSO keyspace-group resolution, GC management, etc.).

### What version of PD are you using (`pd-server -V`)?

N/A — found via code review of `pkg/keyspace/keyspace.go` and `server/server.go`'s bootstrap/startup paths; reproducible on any pre-existing cluster upgrading in place to a PD version where the system keyspace was newly introduced.

Contributor guide

Open the contributing guide

Research direction

Read pkg/keyspace/keyspace.go:172 and the bootstrap/startup paths in server/server.go:819-858. Trace how an already-bootstrapped cluster starts without calling KeyspaceManager.Bootstrap(), then check the system-keyspace assumptions in pkg/tso/keyspace_group_manager.go:84 and pkg/utils/tsoutil/tso_request.go:66. Done means an in-place upgrade creates or backfills the reserved SYSTEM keyspace without requiring a fresh cluster bootstrap.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.