Prevent shard merges which create untrackable shards
Open
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Merging more than 100 shards into one shard makes the final shard untrackable by the data distribution tracker until the data distribution queue executes the merge. (The WaitStorageMetricsPenalty trace event lets you know when this is happening.)
An untrackable shard is undesirable, because we cannot split it if it becomes hot. We should prevent the data distribution tracker from merging more than 100 shards.
Implementing this will be a little tricky, because currently the tracker does not have any information about what the queue is doing. The easiest way is probably to return the number of shards from waitStorageMetrics.
Contributor guide
Assessment
This issue has not been assessed yet.