cockroachdb / cockroachdb/cockroach

rpc: experiment with different codecs (compression algorithms)

Open
#136,563 2 comments 0 reactions 0 assignees View on GitHub
branch-master C-performance o-perf-efficiency
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We currently use snappy compression for our gRPC RPCs. This decision was made years ago, and it might be worth revisiting it with the goal to see if it could be beneficial to switch to another codec. It's likely that snappy is a good choice, but worth a look anyway. Archaeology:

- https://github.com/cockroachdb/cockroach/pull/638
- https://github.com/cockroachdb/cockroach/pull/14239

Uber had an interesting post[^1] for a model serving workload. While likely not representative of our workloads (it looks like they were serving large blobs), it's a good introduction to the topic.

I think in principle we can also switch codecs based on the payload size (and expected response size) but that's likely overkill.

[^1]: https://careersatdoordash.com/blog/enabling-efficient-machine-learning-model-serving/

Jira issue: CRDB-45130

Contributor guide

Open the contributing guide

Research direction

Start by reading the two linked historical pull requests and the referenced workload discussion to understand the existing snappy choice. Locate the gRPC compression path and establish representative benchmarks for alternative codecs; the work is complete when measured results support retaining or changing the codec choice.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend-api-design, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.