cockroachdb / cockroachdb/cockroach
drpc: evaluate an adaptive compression variant
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Goal**
Introduce something like `drpc.CompressionAdaptive` that selects the codec per message by payload size and then benchmark it against snappy and minlz-fastest to decide if it earns its place.
**Design**
- New enum CompressionAdaptive Compression = 3 in drpc.go.
- adaptiveCompress(dst, src, level) in drpcwire/compress.go: len(src) <= 4KiB or len(src) > minlz.MaxBlockSize → snappy.Encode; otherwise minlz.Encode.
- Compress/Decompress/CompressionName/CompressionFromName gain the adaptive case.
- Decompress must use minlz.Decode, not snappy.Decode — it's the superset decoder and transparently handles snappy blocks. This is the linchpin that lets a single wire ID carry mixed encodings; the receiver never needs to know which codec the sender picked.
**Benchmarking**
- Run BenchmarkCompress/BenchmarkDecompress across the corpus (all kinds × sizes) on linux/amd64 (per prior note, amd64 has the full MinLZ assembly path; arm64 results are biased — snappy has full arm64 asm, MinLZ falls back to pure-Go).
- Compare adaptive vs snappy vs minlz-fastest on throughput and ratio, especially around the 4 KiB boundary, to validate the crossover threshold (4 KiB is a guess — the benchmarks should tune it).
**Acceptance**
- adaptive roundtrips correctly for all sizes and both codecs on amd64 and arm64.
- Benchmark data on linux/amd64 either justifies keeping adaptive (and confirms/retunes the threshold) or documents that the win is too small to warrant a third variant, in which case we drop it.
Jira issue: CRDB-66779
Contributor guide
Research direction
Start with drpc.go and drpcwire/compress.go, then locate BenchmarkCompress and BenchmarkDecompress and the existing snappy and minlz-fastest cases. Run the corpus benchmarks on linux/amd64, checking behavior around 4 KiB and round trips on amd64 and arm64. Done means the data either supports and tunes the adaptive variant or documents why it should be dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100