cockroachdb / cockroachdb/cockroach

kvserver: optimize raft entry encoding

Open
#134,936 2 comments 0 reactions 0 assignees View on GitHub
A-kv-replication branch-master C-performance o-perf-efficiency P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We currently store raft entries in nested protobufs[^1] (`RaftMessageRequest` contains `raftpb.Message` contains `[]Entry` contains in its `Data []byte` field (after stripping a prefix) a `kvserverpb.RaftCommand`[^2]. This is all wildly inefficient, I think when we looked at it we found a ~12x allocation overhead (i.e. for one useful byte we allocated 12x as much).

There are various ways to improve this, but one could be adding a new entry encoding in which we use a flatbuffer instead of `kvserverpb.RaftCommand` first (which removes the nested unmarshal), and later possibly replace `RaftMessageRequest` with a flatbuffer as well. However, unless we also use a flatbuffer codec for gRPC, we'll always do some inefficient unmarshaling.

[^2]: https://github.com/cockroachdb/cockroach/blob/90b4bf0e0ea60145c9d759897f612ef7a277d113/pkg/kv/kvserver/raftlog/entry.go#L198

[^1]: https://github.com/cockroachdb/cockroach/blob/90b4bf0e0ea60145c9d759897f612ef7a277d113/pkg/kv/kvserver/kvserverpb/raft.proto#L66

Epic CRDB-55523
Jira issue: CRDB-44302

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.