Ratekeeper allows one storage server to fall behind, even if it is the only replica remaining
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
In order to tolerate isolated slow processes/machines, ratekeeper allows all of the storage processes in 1 fault domain to let their queues grow larger than 1GB (assuming 2+ replication). If this reaches 1.5GB, the storage server will hit the e-brake and start lagging behind the logs.
This process does not take into account the current replication health of the cluster, so if all but one replica of some data dies, ratekeeper may still allow the storage server with the last replica to fall behind. It's also possibly more likely that it would fall behind in this circumstance given that it now has to serve all reads and participate in re-replicating the lost data.
Contributor guide
Assessment
This issue has not been assessed yet.