cockroachdb / cockroachdb/cockroach
kvserver: consider optimizing bulk Raft log application
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When log application falls far behind commits, we may be able to optimize bulk ingestion of these entries. @petermattis says in https://github.com/cockroachdb/cockroach/issues/97044#issuecomment-1462249000:
>Random idea: can we optimize application of a large number of Raft log entries? I believe log entries are applied via individual (unindexed?) Pebble batches. There would be some benefit to using fewer Pebble batches. An even bigger benefit if we could figure out how to convert the log entry application into an sstable to be ingested. That feels hard because log entry application might be overwriting keys written by earlier log entries. There might be something doable here by providing a way to transform a Pebble batch into an sstable, omitting overwrites. The idea here might be a non-starter. Feels worthwhile to loop in some Storage folks to think about the possibilities here.
I believe we already use a single batch for all log entries provided by a Raft ready, which accumulates writes from the individual log entry Pebble batches:
https://github.com/cockroachdb/cockroach/blob/fa17185faa53b535b18e377945275e791ba17253/pkg/kv/kvserver/app_batch.go#L145-L147
This is just a regular indexed batch. It isn't immediately clear to me whether we can trivially change this to an unindexed batch, but worth a try.
https://github.com/cockroachdb/cockroach/blob/554a5d5e28b205d78242d23bc827e60f8033453c/pkg/kv/kvserver/replica_application_state_machine.go#L143
SST ingestion would likely only be worthwhile at larger sizes (tens of megabytes).
This scenario may become more relevant with #94854, although if we separate out the log engine and don't fsync application then it's unclear whether large unapplied log tails would happen that often in practice, since the commit fsync would presumably throttle commits sufficiently for application to catch up.
@pavelkalinnikov @cockroachdb/storage want to see if we have any quick wins here?
Jira issue: CRDB-25225
Contributor guide
Assessment
This issue has not been assessed yet.