keyspace: ensure cache updates are atomic with txn commit
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### Problem
`CreateKeyspace()` performs keyspace creation through multiple independent steps instead of one atomic storage transaction. If an earlier transaction succeeds but a later transaction fails, persistent storage can be left in a partially committed and inconsistent state.
### Affected flow
In `pkg/keyspace/keyspace.go:229`, `CreateKeyspace()` roughly does:
1. `saveNewKeyspace()`
- stores keyspace ID mapping
- stores keyspace metadata in `DISABLED` state
2. `splitKeyspaceRegion()`
3. `UpdateKeyspaceStateByID()`
- updates keyspace state to `ENABLED` in another txn
4. `UpdateKeyspaceForGroup()`
- updates keyspace group membership separately
These steps are not atomic as a whole.
### Current behavior
If `txn1` succeeds but a later step fails, the earlier storage writes remain committed.
Examples:
- keyspace ID and metadata are persisted, but state is never updated to `ENABLED`
- keyspace metadata exists, but keyspace group membership is missing
- region split side effects and storage metadata can observe different progress
This leaves storage in an intermediate state that does not represent either:
- a fully created keyspace, or
- a fully rolled-back keyspace
### Expected behavior
Keyspace creation should provide atomic semantics at the workflow level:
- either all required storage-visible creation steps succeed
- or no partial keyspace state remains persisted
At minimum, failure in later steps should trigger reliable compensation so storage remains consistent.
### Why this matters
A partially created keyspace can confuse later reads and management operations because different subsystems observe different creation progress.
This can cause:
- keyspace metadata existing in storage but not being usable
- missing or incomplete keyspace group membership
- follow-up operations acting on half-created keyspaces
- harder recovery and operational debugging
### Suggested fix
Ensure the create flow has workflow-level atomicity.
Possible directions:
- reduce the number of separate storage transactions where possible
- make intermediate states explicit and safely recoverable
- add compensation/rollback for any already-committed storage updates when later steps fail
- guarantee that keyspace metadata, state transition, and group membership cannot be left inconsistent after failure
### Reproduction idea
Inject a failure after `saveNewKeyspace()` succeeds but before `UpdateKeyspaceStateByID()` or `UpdateKeyspaceForGroup()` completes.
Then verify storage contains partial keyspace data, such as:
- keyspace ID mapping exists
- keyspace metadata exists in `DISABLED` state or incomplete state
- keyspace group assignment is missing
This demonstrates that one transaction can succeed while a later one fails, leaving inconsistent persistent state.
Contributor guide
Assessment
This issue has not been assessed yet.