Improve FDB reliability when localities are misconfigured and later corrected
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
If a storage server (SS) does not have a valid locality due to misconfiguration, the replication policy can select replicas that do not satisfy the policy. For example, in `three_data_hall` mode, if a server is not configured with `data_hall` locality, the `selectReplicas()` may create a server team whose size is not equal to the replica factor.
Another issue is that a SS may be chosen as the preferred server and feed into `selectReplicas()`, although having the SS will never create a valid team. For example, in `three_data_center` mode, if a DC has only one server and the server is chosen as the must-have one for a team. `selectReplicas()` will not be able to create such a team. `addTeamsBest()` may get stuck there.
Although this only happens in misconfiguration, DD should better prevent itself from the problem using the following solution:
1) If a SS does not have a valid locality configuration under a replication policy, it should not be used in building teams -- it should be treated as always unhealthy. It should also create a trace event to notify the system operator;
2) We should add test cases in simulation to cover these situations: `three_data_hall` mode, and `three_data_center` mode.
Contributor guide
Research direction
Start by tracing selectReplicas() and addTeamsBest() in the data distribution (DD) code, focusing on invalid localities and preferred servers. Review the simulation coverage for three_data_hall and three_data_center. Done means invalid-locality servers are excluded from teams, a trace event is emitted, and both misconfiguration scenarios are covered by simulation tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100