cockroachdb / cockroachdb/cockroach

kvserver: flush Raft log to OS with `kv.raft_log.disable_synchronization_unsafe`

Open
#104,724 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

See cockroachdb/pebble#2624. Quoting from there:

>Committing a batch with Sync: false not only disables fsync, but also disables flushing the write to the OS. This means that it's not only vulnerable to data loss on OS crashes or power loss, but also on unclean process termination, which is a much more common failure.
>
>This is particularly problematic with e.g. Raft replication, where some users may want to disable fsync for the performance gain, accepting the risk of data loss when a quorum of replicas lose writes with the reasoning that correlated power loss or OS crashes across failure domains is rare. However, correlated process crashes is much higher probability: consider e.g. a panic or OOM that's triggered by user queries, where these are load balanced across several nodes holding replicas of the range.
>
>To mitigate this risk, we should either make Sync: false flush the write to the OS, or add another setting that enables this behavior. This may imply slightly lower performance, but that seems worth the significant reliability improvement.

See also [internal discussion](https://cockroachlabs.slack.com/archives/C05CKCM4Z08/p1686566623718579).

Jira issue: CRDB-28694

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.