tikv / tikv/pd

keyspace: batch removal from a TSO keyspace group can exceed etcd limits

Open
#11,169 0 comments 0 reactions 0 assignees View on GitHub
severity/moderate type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Send one request containing a large number of archived or tombstone
keyspace IDs to:

`DELETE /pd/api/v2/tso/keyspace-groups/{group_id}/keyspaces`

This operation removes the specified keyspaces from the TSO keyspace
group. It is separate from a complete keyspace teardown workflow.

### What did you expect to see?

Large removal requests should complete without exceeding the etcd
transaction operation limit.

The operation should use bounded, retry-safe transactions while keeping
the keyspace group membership and the metadata changes performed by this
API consistent.

Meta-service group assignment counts should also remain accurate when
multiple removed keyspaces belong to the same group.

### What did you see instead?

`GroupManager.RemoveKeyspacesFromGroup` currently processes the entire
request in one storage transaction.

For each accepted keyspace, the current implementation schedules deletion
of its metadata and name-to-ID mapping. The transaction also saves the
updated TSO keyspace group.

Without meta-service group assignments, removing 64 keyspaces therefore
schedules at least 129 writes:

`64 * 2 + 1 = 129`

This exceeds etcd's default limit of 128 operations per transaction.

Archived keyspaces with meta-service group assignments can add another
status update per keyspace. Multiple keyspaces assigned to the same
meta-service group may also schedule repeated writes to the same status
key, leaving the persisted assignment count inaccurate.

This issue only covers batching and consistency for the TSO keyspace
group removal API. Complete keyspace teardown, including Resource Manager
and other keyspace-scoped metadata, is a separate problem.

### What version of PD are you using (`pd-server -V`)?

Current master at commit
`a77df243d98b9852abfa5649ba4b4612253d71a3`.

Contributor guide

Open the contributing guide

Research direction

Start at the DELETE /pd/api/v2/tso/keyspace-groups/{group_id}/keyspaces endpoint and GroupManager.RemoveKeyspacesFromGroup. Trace the storage transaction and meta-service assignment updates, then verify bounded retry-safe batches, consistent group membership metadata, and accurate counts when several removed keyspaces share a group.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.