cockroachdb / cockroachdb/cockroach

drpc: evaluate an adaptive compression variant

Open
#173,462 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Goal**
Introduce something like `drpc.CompressionAdaptive` that selects the codec per message by payload size and then benchmark it against snappy and minlz-fastest to decide if it earns its place.

**Design**
- New enum CompressionAdaptive Compression = 3 in drpc.go.
- adaptiveCompress(dst, src, level) in drpcwire/compress.go: len(src) <= 4KiB or len(src) > minlz.MaxBlockSize → snappy.Encode; otherwise minlz.Encode.
- Compress/Decompress/CompressionName/CompressionFromName gain the adaptive case.
- Decompress must use minlz.Decode, not snappy.Decode — it's the superset decoder and transparently handles snappy blocks. This is the linchpin that lets a single wire ID carry mixed encodings; the receiver never needs to know which codec the sender picked.

**Benchmarking**
- Run BenchmarkCompress/BenchmarkDecompress across the corpus (all kinds × sizes) on linux/amd64 (per prior note, amd64 has the full MinLZ assembly path; arm64 results are biased — snappy has full arm64 asm, MinLZ falls back to pure-Go).
- Compare adaptive vs snappy vs minlz-fastest on throughput and ratio, especially around the 4 KiB boundary, to validate the crossover threshold (4 KiB is a guess — the benchmarks should tune it).

**Acceptance**
- adaptive roundtrips correctly for all sizes and both codecs on amd64 and arm64.
- Benchmark data on linux/amd64 either justifies keeping adaptive (and confirms/retunes the threshold) or documents that the win is too small to warrant a third variant, in which case we drop it.

Jira issue: CRDB-66779

Contributor guide

Open the contributing guide

Research direction

Start with drpc.go and drpcwire/compress.go, then locate BenchmarkCompress and BenchmarkDecompress and the existing snappy and minlz-fastest cases. Run the corpus benchmarks on linux/amd64, checking behavior around 4 KiB and round trips on amd64 and arm64. Done means the data either supports and tunes the adaptive variant or documents why it should be dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.