apple / apple/foundationdb

Don't register new StorageServerInterface on startup until Storage is caught up

Open
#6,321 3 comments 0 reactions 1 assignee Assigned to @sfc-gh-bvr View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Currently, when a storage server restarts it immediately will register itself in the cluster. This can cause many problems, especially if the storage server has been down for a while:

* Ratekeeper could go down if multiple storages are brought back together because all of them will be behind (and therefore will have a large storage queue).
* Clients will potentially try to read from a storage server that is far behind. This causes high latency on the client and uses CPU cycles on the storage (and those cycles would be better used to catch up).
* Worst case a storage won't be able to catch up because of the above affects and we have to rely on ratekeeper.

Not registering the storage will also have an effect that data distribution will still do self-healing until the storage has caught up. This is probably a good thing, as this will allow the cluster to repair even if there's a slow disk with a flaky storage server.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.