apple / apple/foundationdb

Data distribution rebalancing sometimes stops in an unbalanced state

Open
#1,884 0 comments 4 reactions 0 assignees View on GitHub
data distribution
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I've seen a few situations where a cluster showed a notable disparity in the bytes stored distribution across its storage nodes, and data movement would rebalance in order to resolve that disparity. Then for reasons I have not investigated, data movement would stop prior to parity being restored.

Bouncing the cluster seemed to get data movement rolling again, and it would make more progress before eventually stopping.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reproducing the issue with a cluster that has uneven bytes across storage nodes, then trace why data movement stops and whether restarting resumes it. Done means rebalancing reliably reaches parity without requiring a cluster restart, with coverage for the stopping case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.