tikv / tikv/pd

keyspace: ensure cache updates are atomic with txn commit

Open
#10,461 0 comments 0 reactions 0 assignees View on GitHub
affects-8.5 severity/minor type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### Problem

`CreateKeyspace()` performs keyspace creation through multiple independent steps instead of one atomic storage transaction. If an earlier transaction succeeds but a later transaction fails, persistent storage can be left in a partially committed and inconsistent state.

### Affected flow

In `pkg/keyspace/keyspace.go:229`, `CreateKeyspace()` roughly does:

1. `saveNewKeyspace()`
- stores keyspace ID mapping
- stores keyspace metadata in `DISABLED` state
2. `splitKeyspaceRegion()`
3. `UpdateKeyspaceStateByID()`
- updates keyspace state to `ENABLED` in another txn
4. `UpdateKeyspaceForGroup()`
- updates keyspace group membership separately

These steps are not atomic as a whole.

### Current behavior

If `txn1` succeeds but a later step fails, the earlier storage writes remain committed.

Examples:
- keyspace ID and metadata are persisted, but state is never updated to `ENABLED`
- keyspace metadata exists, but keyspace group membership is missing
- region split side effects and storage metadata can observe different progress

This leaves storage in an intermediate state that does not represent either:
- a fully created keyspace, or
- a fully rolled-back keyspace

### Expected behavior

Keyspace creation should provide atomic semantics at the workflow level:
- either all required storage-visible creation steps succeed
- or no partial keyspace state remains persisted

At minimum, failure in later steps should trigger reliable compensation so storage remains consistent.

### Why this matters

A partially created keyspace can confuse later reads and management operations because different subsystems observe different creation progress.

This can cause:
- keyspace metadata existing in storage but not being usable
- missing or incomplete keyspace group membership
- follow-up operations acting on half-created keyspaces
- harder recovery and operational debugging

### Suggested fix

Ensure the create flow has workflow-level atomicity.

Possible directions:
- reduce the number of separate storage transactions where possible
- make intermediate states explicit and safely recoverable
- add compensation/rollback for any already-committed storage updates when later steps fail
- guarantee that keyspace metadata, state transition, and group membership cannot be left inconsistent after failure

### Reproduction idea

Inject a failure after `saveNewKeyspace()` succeeds but before `UpdateKeyspaceStateByID()` or `UpdateKeyspaceForGroup()` completes.

Then verify storage contains partial keyspace data, such as:
- keyspace ID mapping exists
- keyspace metadata exists in `DISABLED` state or incomplete state
- keyspace group assignment is missing

This demonstrates that one transaction can succeed while a later one fails, leaving inconsistent persistent state.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.