apple / apple/foundationdb

Improve FDB reliability when localities are misconfigured and later corrected

Open
#2,100 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

If a storage server (SS) does not have a valid locality due to misconfiguration, the replication policy can select replicas that do not satisfy the policy. For example, in `three_data_hall` mode, if a server is not configured with `data_hall` locality, the `selectReplicas()` may create a server team whose size is not equal to the replica factor.

Another issue is that a SS may be chosen as the preferred server and feed into `selectReplicas()`, although having the SS will never create a valid team. For example, in `three_data_center` mode, if a DC has only one server and the server is chosen as the must-have one for a team. `selectReplicas()` will not be able to create such a team. `addTeamsBest()` may get stuck there.

Although this only happens in misconfiguration, DD should better prevent itself from the problem using the following solution:
1) If a SS does not have a valid locality configuration under a replication policy, it should not be used in building teams -- it should be treated as always unhealthy. It should also create a trace event to notify the system operator;
2) We should add test cases in simulation to cover these situations: `three_data_hall` mode, and `three_data_center` mode.

Contributor guide

Open the contributing guide

Research direction

Start by tracing selectReplicas() and addTeamsBest() in the data distribution (DD) code, focusing on invalid localities and preferred servers. Review the simulation coverage for three_data_hall and three_data_center. Done means invalid-locality servers are excluded from teams, a trace event is emitted, and both misconfiguration scenarios are covered by simulation tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.