cockroachdb / cockroachdb/cockroach

raft: relax fsync requirements

Open
#124,278 3 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Raft requires fsync on casting votes and accepting entries into the log. All decisions on the critical path towards entry commits must be locally durable. A node restart in this model is indistinguishable from a node being slow.

This has a couple of downsides. For one, fsync latency adds into commit latency.

More critically, in practice, some environments either explicitly disable fsync (to avoid the aforementioned latency), or have system bugs in fsync itself which result in violating this requirement. This leads at best to node crashes in raft (e.g. when the follower state regresses after a restart, and the leader tries to [move it forward](https://github.com/cockroachdb/cockroach/blob/021c72ae4a131bf511bff654fd3e4162947d3b15/pkg/raft/log.go#L329) with out-of-bound updates) and loss of quorum, and in worst cases to busy loops, silent durability loss, or silently uncommitting entries.

Raft's failure model can be relaxed, in order to workaround a minority of nodes having fsync problems. It is sufficient to require a "collective" durability: the state is durable on a quorum, but every individual replica can fall behind on restart. This model is supported in [Viewstamped Replication](https://pmg.csail.mit.edu/papers/vr-revisited.pdf) (sections 4.3, 5.1), and can be adopted in any consensus algorithm.

To support this model, a restarted node must not immediately assume its state has been durable. Instead, it starts as a learner, learns the up-to-date state (or makes sure its state is ok) before proceeding to be a voter. The node can do so by coordinating with a quorum and/or the leader. In a way, a replica restart is equivalent to a permanent death of this replica, and a safe reconfiguration in which its new incarnation rejoins the group.

Jira issue: CRDB-38804

Related/duplicate to #88442
Related to #113147

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.