cockroachdb / cockroachdb/cockroach

kvserver: switch to append-time conf changes

Open
#107,083 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

etcd-io/raft uses apply-time conf changes. These are not what's described in the raft paper, and they are much more complicated to reason about[^1]. They become a mental burden every time we reason about the durability requirements of the `HardState.Commit` index, for example in https://github.com/cockroachdb/cockroach/issues/99273#issuecomment-1622178841) and https://github.com/cockroachdb/cockroach/issues/88699. Even if these problems are ultimately tractable, it's just complex.

The reason it was introduced as is is that it is also convenient to have the active raft config track exactly the RangeDescriptor. With append-time conf changes, we need to track the active config separately (it's the latest one synced in the log) and might need to revert it (if the log gets rewritten). We would also need a bit more observability if the active config leads the range descriptor (which happens while the new config is in the log but not applied yet).

Given how much cognitive trouble we've had with apply-time conf changes, I think these trade-offs are well worth it.

I believe we could use append-time conf changes without changing upstream raft: we stop using `raftpb.ConfChange{,V2}`, instead proposing our conf changes as regular proposals. The fact that they're conf changes instead gets encoded in our `RaftCommandPrefix`. Our code that appends to the log can then sniff them out and update state as appropriate, calling `rawNode.ApplyConfChange`, which looks like it will "just work". We should be able to add appropriate testing upstream to make this a "permissible" use of `etcd-io/raft` (rather than us hacking it in ways that might break in the future).

[^1]: https://github.com/etcd-io/etcd/issues/7625#issuecomment-489232411

I haven't verified this approach above, but it lends itself to prototyping.

Jira issue: CRDB-39895

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.