Optimize GCStateManager for capacity in large keyspace number scenarios (100k - 1M keyspaces)
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
The `GCStateManager`, the core part of the GC refactory project (#8978), was not yet optimized for scenarios for massive keyspaces. However, as the situation and the product roadmap changes, the related optimization becomes a strong requirement. Also, we've encountered some incidents where high concurrency GC-related calls causes problem on PD or etcd, which should be avoided in the future.
Basically, we expect that the `GCStateManager` should be capable to serve at most 1M keyspaces, so that `GCStateManager` won't be the performance bottleneck for a long time.
### Subtasks
- [ ] Support excluding GC barriers for GC state retrieving APIs (`GetGCState`, `GetAllKeyspacesGCStates`)
- [ ] GC state cache (excluding GC barriers)
- Ref: #10607
- [ ] WatchGCState
- Ref: #10497
- [ ] Replace mutex (whose critical section contains IO operations) with cancellable synchronization methods
- [ ] Batch GC state writes
- [ ] Keyspace level mutexes and transactions (instead of global; requires special synchronization algorithm for global GC barriers)
Contributor guide
Assessment
This issue has not been assessed yet.