cockroachdb / cockroachdb/cockroach
cdc, util: use incremental encoding of spans in more places
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Outside of pebble, we tend to serialize lists of spans as just a `[]Span`, meaning a lot of repeated bytes. This reduces, in particular, horizontal scalability because we have to reduce the granularity of span lists to get them to fit into memory. In changefeeds, this means we have giant checkpoint records that still don't fully capture the state, increasing the number of duplicate messages.
Spans lend themselves well to delta/incremental encoding, and much of the work needed for efficient encoding is already being done when building a roachpb.SpanGroup. https://github.com/HonoreDB/cockroach/commit/f65a61d0d7b2cf130d0d1e547e82cdce54fe0fc3 spikes out a simple encoding scheme that's more efficient (in all respects) than standard compression. It might be worth iterating on it to further reduce the delimiter size.
https://github.com/HonoreDB/cockroach/commit/fcc11d1de50d434c4f61894f38bf4b429c366799 is a (buggy, iirc) use of it for CDC checkpoints.
Jira issue: CRDB-29680
Epic CRDB-31494
Contributor guide
Assessment
This issue has not been assessed yet.