apple / apple/foundationdb

Data movement can't begin on clusters missing data

Open
#3,774 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

When a cluster loses all replicas of a shard and the data distributor later restarts, it gets stuck trying to track the initial shards (Note: I'm not certain if this is universally true or if it requires other properties to hold). As a result, no data movement can happen with the data that still exists in the cluster.

It would be better if data movement could continue on the shards that remain, which could help us to prevent increasing the blast radius of this failure case in some circumstances.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Begin by tracing the data distributor's restart and initial-shard tracking path, then reproduce the missing-replica scenario; done means data movement continues for surviving shards after restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.