[nextgen] Cross-keyspace runtime can still mutate DISABLED and ARCHIVED keyspaces after a fresh reopen
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
Use a next-gen TiDB checkout with:
- real PD + real TiKV
- TiKV API v2 enabled
- `go test --tags=intest,nextgen`
Run a focused probe that:
1. creates fresh unique keyspaces `SYSTEM`, ``, and `` in PD;
2. boots a SYSTEM-keyspace domain plus user-keyspace runtimes;
3. creates a user table in each target keyspace;
4. changes one target keyspace to `DISABLED`, and the other to `DISABLED` then `ARCHIVED`;
5. calls `AcquireKSRuntime(targetKS, holderID)` from the SYSTEM-keyspace domain;
6. creates an import job through the fresh cross-keyspace runtime;
7. verifies from a fresh session in the target keyspace that the durable row in
`mysql.tidb_import_jobs` is present.
Observed result on the current next-gen path:
```text
DISABLED:
update keyspace ... to DISABLED
acquire fresh runtime
create import job through cross-keyspace runtime
verify durable job visibility in target keyspace
PASS
ARCHIVED:
update keyspace ... to DISABLED
update keyspace ... to ARCHIVED
acquire fresh runtime
create import job through cross-keyspace runtime
verify durable job visibility in target keyspace
PASS
```
Expected result:
```text
After PD has published DISABLED or ARCHIVED for a keyspace,
fresh TiDB-side cross-keyspace runtime acquisition should fail closed before a store is returned,
so no new import job or other transactional mutation can be created in that keyspace.
```
### 2. What did you expect to see? (Required)
`DISABLED` and `ARCHIVED` keyspaces should reject new cross-keyspace TiKV client construction
and therefore reject fresh SYSTEM-keyspace background work that tries to reopen them by name.
### 3. What did you see instead? (Required)
Even after PD has already moved the target keyspace out of `ENABLED`, a fresh SYSTEM-keyspace
runtime can still reopen that keyspace and create durable state inside it. This is not limited to
the lower-level client-go matrix; a higher-level TiDB consumer can do it too.
### 4. What is your TiDB version? (Required)
- TiDB current next-gen source under local checkout
- client-go dependency path still uses `GetKeyspaceMeta` for `NewCodecPDClientWithKeyspace`
- real PD/TiKV with API v2 enabled for the test
## Why this happens
At the dependency boundary:
- `GetKeyspaceID` loads keyspace metadata and explicitly rejects states other than `ENABLED`.
- `GetKeyspaceMeta` loads the same metadata but returns all states unchanged.
- `NewCodecPDClientWithKeyspace` uses `GetKeyspaceMeta`, so it preserves identity but drops
the lifecycle admission predicate.
At the TiDB consumer boundary:
- SYSTEM-keyspace cross-keyspace runtime creation uses `InitStorage(targetKS)`;
- that path constructs a fresh API-v2 client by keyspace name;
- DXF schedulers/executors and other SYSTEM-keyspace consumers can reach it after reload/restart;
- the focused probe shows a fresh SYSTEM-keyspace consumer can mutate `mysql.tidb_import_jobs`
in a target keyspace after PD has already published `DISABLED` or `ARCHIVED`.
So a fresh reopen after state publication still succeeds, and higher-level TiDB background work can
mutate a keyspace PD has already marked `DISABLED` or `ARCHIVED`.
## Impact
This breaks the lifecycle boundary that non-ENABLED keyspace states are supposed to provide.
A delayed or restarted SYSTEM-keyspace background consumer can continue mutating a keyspace after
the control plane has already retired it from normal service.
## Fix direction
Restore the missing lifecycle gate on the fresh-open path, for example by enforcing the same
`meta.State == ENABLED` predicate in the metadata-returning helper used by
`NewCodecPDClientWithKeyspace`, or by adding an equivalent fail-closed admission check before
cross-keyspace runtime creation returns a usable store.
Contributor guide
Research direction
Start with GetKeyspaceMeta and NewCodecPDClientWithKeyspace, then trace SYSTEM-keyspace cross-keyspace runtime creation through InitStorage. Run the focused probe with go test --tags=intest,nextgen against real PD and TiKV. Done means fresh acquisition rejects DISABLED and ARCHIVED keyspaces before returning a usable store, with the probe confirming no durable import job is created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100