apple / apple/foundationdb

Prevent shard merges which create untrackable shards

Open
#2,541 0 comments 0 reactions 1 assignee Claimed by @etschannen View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Merging more than 100 shards into one shard makes the final shard untrackable by the data distribution tracker until the data distribution queue executes the merge. (The WaitStorageMetricsPenalty trace event lets you know when this is happening.)

An untrackable shard is undesirable, because we cannot split it if it becomes hot. We should prevent the data distribution tracker from merging more than 100 shards.

Implementing this will be a little tricky, because currently the tracker does not have any information about what the queue is doing. The easiest way is probably to return the number of shards from waitStorageMetrics.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.