cockroachdb / cockroachdb/cockroach

kvserver: investigate exceeding uncommitted entry size

Open
#100,096 11 comments 0 reactions 0 assignees View on GitHub
branch-master branch-release-23.1 C-investigation T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

See https://github.com/cockroachdb/cockroach/issues/99464. We were seeing lots of these messages:

> appending new entries to log would exceed uncommitted entry size limit; dropping proposal

That's not good! These proposals will have to be picked up by `refreshProposalsLocked` which takes ~200ms.

If something changed recently that causes us to hit this more frequently (as seems to be the case, anecdotally) we need to add at least rudimentary flow control here.

It is somewhat expected to hit this in general. To generate proposals and to put them into raft requires only disk reads, but getting them committed requires a durable write, including likely round-trips to followers (to obtain quorum). The former can be done at much higher throughput than the latter.

**To Reproduce**

Running the test in #99464 reproduces this in 100% of all runs I did, averaging around ~70 related log messages per run.

~~I also seem to hit this reliably with the unit test `BenchmarkReplicaProposal`~~ (edit: not true, we're dropping proposals but not becaue of uncommitted entry size but because of no leader)

**Expected behavior**

Some kind of flow control, similar to the quota pool but locally. Ideally we would delay grabbing a latch for a write if we can tell that raft is not currently accepting more proposals. Maybe we want to avoid doing most of this buffering inside of RawNode anyway, and instead improve our proposal buffer to provide this backpressure signal instead. Unsure, needs more thought.

**Additional data / screenshots**

Adding better visibility into this issue in https://github.com/cockroachdb/cockroach/pull/100083.

Jira issue: CRDB-26247

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.