cockroachdb / cockroachdb/cockroach

kv: make disk reads asynchronous with respect to Raft state machine

Open
#105,850 8 comments 2 reactions 0 assignees View on GitHub
A-kv-replication C-performance T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue is the "disk read" counterpart to https://github.com/cockroachdb/cockroach/issues/17500, which was addressed by https://github.com/etcd-io/raft/pull/8 and https://github.com/cockroachdb/cockroach/pull/94165. To contextualize this issue, it may be helpful to get re-familiarized with those, optionally with [this presentation](https://docs.google.com/presentation/d/1owtj5S38Qky8yWz91sfs7x0pkbrLYTwhSlSJ3QMvADw/edit?usp=sharing).

The raft state machine loop (`handleRaftReady`) is responsible for writing raft entries to the durable raft log, applying committed log entries to the state machine, and sending messages to peers. This event loop is the heart of the raft protocol and each raft write traverses it multiple times between proposal time and ack time. It is therefore important to keep the latency of this loop down, so that a slow iteration does not block writes in the pipeline and create cross-write interference.

To that end, https://github.com/cockroachdb/cockroach/pull/94165 made raft log writes non-blocking in this loop, so that slow log writes (which much fsync) do not block other raft proposals.

Another case where the event loop may synchronously touch disk is when constructing the list of committed entries to apply. In the common case, this [pulls from the raft entry cache](https://github.com/cockroachdb/cockroach/blob/9b1753366e0307c83eb59a6351a844467b6cf9d3/pkg/kv/kvserver/logstore/logstore.go#L522), so it is fast. However, on raft entry cache misses, this [reads from pebble](https://github.com/cockroachdb/cockroach/blob/9b1753366e0307c83eb59a6351a844467b6cf9d3/pkg/kv/kvserver/logstore/logstore.go#L573). Reads from pebble can be slow (relative to a cache hit), which can slow down the event loop because they are performed inline. The effect of this can be seen directly on raft scheduling tail latency.

Example graphs

### Entry cache hit rates

Screenshot 2023-06-29 at 2 09 16 AM

| | Accesses | Hits | Hit Rate |
| ----- | -------- | ------ | -------- |
| n1 | 314468 | 308334 | 98.1% |
| n2 | 276748 | 260645 | 94.2% |
| n3 | 271915 | 255306 | 93.9% |
| n4 | 325052 | 320766 | 98.7% |
| n5 | 326403 | 321934 | 98.6% |

### Raft scheduler latencies

Screenshot 2023-06-29 at 2 13 39 AM

### High raft entry cache hit rate (n4)

Screenshot 2023-06-29 at 2 44 13 AM

### Low raft entry cache hit rate (n3)

Screenshot 2023-06-29 at 2 04 40 AM

An alternate design would be to make these disk reads async on raft entry cache misses. Instead of blocking on the log iteration, `raft.Storage.Entries` could support returning a new `ErrEntriesTemporarilyUnavailable` error which instructs etcd/raft to retry the read later. This would allow the event loop to continue processing. When the read completes, the event loop would be notified and the read would be retried from the cache (or some other data structure that has no risk of eviction before the read is retries).

This would drive down tail latency for raft writes in cases where the raft entry cache has a less than perfect hit rate.

Jira issue: CRDB-29234

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.