apple / apple/foundationdb

Add simulation test to ensure each storage server has similar space utilization

Open
#1,766 4 comments 0 reactions 0 assignees View on GitHub
data distribution
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We want each storage server has similar space utilization for balancing the disk space pressure among storage servers. For example, if each storage server has the same amount of space, we want each storage server has similar amount (e.g., +-5% difference) of data.

The current implementation only test that there is no data movement when a cluster reaches to a stable state. However, it is possible that the cluster only uses a set of storage servers and balance the data on them, leaving some storage servers empty.

We can add the test in `waitForQuietDatabase()` to check the shards size on each storage server, and make sure the standard deviation is not too big (say within +-5% of the average size).

Contributor guide

Open the contributing guide

Research direction

Start by reading waitForQuietDatabase() and the existing simulation test that checks for no data movement after the cluster stabilizes. Inspect how shard sizes are obtained for each storage server, then add the requested utilization check and verify that the test rejects distributions outside the stated +-5% average range.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.