cockroachdb / cockroachdb/cockroach

kvserver: Raft processing should handle data loss/corruption gracefully

Open
#113,147 2 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, both etcd/raft and CRDB itself will tend to exit with a fatal error if Raft data has been lost or corrupted. This is often seen when running without fsync, either at the OS level or via `kv.raft_log.synchronization.unsafe.disabled`. The typical failure is e.g. Raft fataling with "state.commit 267 is out of range [523, 728]" or CRDB fatal application failures (#75944), but there is a large number of other fatal errors both in Raft and CRDB, and there are also other failure modes such as append loops (#113053). In principle, there's an unbounded number of failure modes here, given there's an unbounded number of ways data could become lost or corrupted.

We should try to handle these failures more gracefully. Ideally, we would want to isolate the failure to a single replica instead of failing the entire node, cordoning it off and having the allocator upreplicate elsewhere (assuming the entire range isn't faulty). This is a generalization of the planned handling of application failures (#75944).

This is a pre-requisite for allowing users to safely disable fsync (#88442) -- at the very least, we need to detect when a node has lost data and provide a user-friendly error message.

Jira issue: CRDB-32773

Contributor guide

Open the contributing guide

Research direction

Start by reading the kvserver Raft processing and etcd/raft failure paths described in the issue, including the out-of-range commit and append-loop cases; no specific file or test is named. Done means detecting lost or corrupted data, isolating the affected replica rather than failing the node, and reporting a user-friendly error so the allocator can upreplicate elsewhere.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.