cockroachdb / cockroachdb/cockroach
kvserver: flush Raft log to OS with `kv.raft_log.disable_synchronization_unsafe`
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
See cockroachdb/pebble#2624. Quoting from there:
>Committing a batch with Sync: false not only disables fsync, but also disables flushing the write to the OS. This means that it's not only vulnerable to data loss on OS crashes or power loss, but also on unclean process termination, which is a much more common failure.
>
>This is particularly problematic with e.g. Raft replication, where some users may want to disable fsync for the performance gain, accepting the risk of data loss when a quorum of replicas lose writes with the reasoning that correlated power loss or OS crashes across failure domains is rare. However, correlated process crashes is much higher probability: consider e.g. a panic or OOM that's triggered by user queries, where these are load balanced across several nodes holding replicas of the range.
>
>To mitigate this risk, we should either make Sync: false flush the write to the OS, or add another setting that enables this behavior. This may imply slightly lower performance, but that seems worth the significant reliability improvement.
See also [internal discussion](https://cockroachlabs.slack.com/archives/C05CKCM4Z08/p1686566623718579).
Jira issue: CRDB-28694
Contributor guide
Assessment
This issue has not been assessed yet.