tikv / tikv/pd

Optimize GCStateManager for capacity in large keyspace number scenarios (100k - 1M keyspaces)

Open
#10,659 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

The `GCStateManager`, the core part of the GC refactory project (#8978), was not yet optimized for scenarios for massive keyspaces. However, as the situation and the product roadmap changes, the related optimization becomes a strong requirement. Also, we've encountered some incidents where high concurrency GC-related calls causes problem on PD or etcd, which should be avoided in the future.

Basically, we expect that the `GCStateManager` should be capable to serve at most 1M keyspaces, so that `GCStateManager` won't be the performance bottleneck for a long time.

### Subtasks

- [ ] Support excluding GC barriers for GC state retrieving APIs (`GetGCState`, `GetAllKeyspacesGCStates`)
- [ ] GC state cache (excluding GC barriers)
- Ref: #10607
- [ ] WatchGCState
- Ref: #10497
- [ ] Replace mutex (whose critical section contains IO operations) with cancellable synchronization methods
- [ ] Batch GC state writes
- [ ] Keyspace level mutexes and transactions (instead of global; requires special synchronization algorithm for global GC barriers)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.