apple / apple/foundationdb

Ratekeeper allows one storage server to fall behind, even if it is the only replica remaining

Open
#493 0 comments 0 reactions 1 assignee Claimed by @ajbeamon View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In order to tolerate isolated slow processes/machines, ratekeeper allows all of the storage processes in 1 fault domain to let their queues grow larger than 1GB (assuming 2+ replication). If this reaches 1.5GB, the storage server will hit the e-brake and start lagging behind the logs.

This process does not take into account the current replication health of the cluster, so if all but one replica of some data dies, ratekeeper may still allow the storage server with the last replica to fall behind. It's also possibly more likely that it would fall behind in this circumstance given that it now has to serve all reads and participate in re-replicating the lost data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.