pingcap / pingcap/tidb

[nextgen] Cross-keyspace runtime can still mutate DISABLED and ARCHIVED keyspaces after a fresh reopen

Open
#70,592 1 comment 0 reactions 0 assignees View on GitHub
component/tikv-client found-by-ai may-affects-25.10 may-affects-26.3 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Use a next-gen TiDB checkout with:

- real PD + real TiKV
- TiKV API v2 enabled
- `go test --tags=intest,nextgen`

Run a focused probe that:

1. creates fresh unique keyspaces `SYSTEM`, ``, and `` in PD;
2. boots a SYSTEM-keyspace domain plus user-keyspace runtimes;
3. creates a user table in each target keyspace;
4. changes one target keyspace to `DISABLED`, and the other to `DISABLED` then `ARCHIVED`;
5. calls `AcquireKSRuntime(targetKS, holderID)` from the SYSTEM-keyspace domain;
6. creates an import job through the fresh cross-keyspace runtime;
7. verifies from a fresh session in the target keyspace that the durable row in
`mysql.tidb_import_jobs` is present.

Observed result on the current next-gen path:

```text
DISABLED:
update keyspace ... to DISABLED
acquire fresh runtime
create import job through cross-keyspace runtime
verify durable job visibility in target keyspace
PASS

ARCHIVED:
update keyspace ... to DISABLED
update keyspace ... to ARCHIVED
acquire fresh runtime
create import job through cross-keyspace runtime
verify durable job visibility in target keyspace
PASS
```

Expected result:

```text
After PD has published DISABLED or ARCHIVED for a keyspace,
fresh TiDB-side cross-keyspace runtime acquisition should fail closed before a store is returned,
so no new import job or other transactional mutation can be created in that keyspace.
```

### 2. What did you expect to see? (Required)

`DISABLED` and `ARCHIVED` keyspaces should reject new cross-keyspace TiKV client construction
and therefore reject fresh SYSTEM-keyspace background work that tries to reopen them by name.

### 3. What did you see instead? (Required)

Even after PD has already moved the target keyspace out of `ENABLED`, a fresh SYSTEM-keyspace
runtime can still reopen that keyspace and create durable state inside it. This is not limited to
the lower-level client-go matrix; a higher-level TiDB consumer can do it too.

### 4. What is your TiDB version? (Required)

- TiDB current next-gen source under local checkout
- client-go dependency path still uses `GetKeyspaceMeta` for `NewCodecPDClientWithKeyspace`
- real PD/TiKV with API v2 enabled for the test

## Why this happens

At the dependency boundary:

- `GetKeyspaceID` loads keyspace metadata and explicitly rejects states other than `ENABLED`.
- `GetKeyspaceMeta` loads the same metadata but returns all states unchanged.
- `NewCodecPDClientWithKeyspace` uses `GetKeyspaceMeta`, so it preserves identity but drops
the lifecycle admission predicate.

At the TiDB consumer boundary:

- SYSTEM-keyspace cross-keyspace runtime creation uses `InitStorage(targetKS)`;
- that path constructs a fresh API-v2 client by keyspace name;
- DXF schedulers/executors and other SYSTEM-keyspace consumers can reach it after reload/restart;
- the focused probe shows a fresh SYSTEM-keyspace consumer can mutate `mysql.tidb_import_jobs`
in a target keyspace after PD has already published `DISABLED` or `ARCHIVED`.

So a fresh reopen after state publication still succeeds, and higher-level TiDB background work can
mutate a keyspace PD has already marked `DISABLED` or `ARCHIVED`.

## Impact

This breaks the lifecycle boundary that non-ENABLED keyspace states are supposed to provide.
A delayed or restarted SYSTEM-keyspace background consumer can continue mutating a keyspace after
the control plane has already retired it from normal service.

## Fix direction

Restore the missing lifecycle gate on the fresh-open path, for example by enforcing the same
`meta.State == ENABLED` predicate in the metadata-returning helper used by
`NewCodecPDClientWithKeyspace`, or by adding an equivalent fail-closed admission check before
cross-keyspace runtime creation returns a usable store.

Contributor guide

Open the contributing guide

Research direction

Start with GetKeyspaceMeta and NewCodecPDClientWithKeyspace, then trace SYSTEM-keyspace cross-keyspace runtime creation through InitStorage. Run the focused probe with go test --tags=intest,nextgen against real PD and TiKV. Done means fresh acquisition rejects DISABLED and ARCHIVED keyspaces before returning a usable store, with the probe confirming no durable import job is created.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.