bazelbuild / bazelbuild/bazel

[FeatureRequest] Support ZSTD encoding for HTTP remote cache server

Open
#23,463 0 comments 0 reactions 0 assignees View on GitHub
help wanted P3 team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

The common remote-cache server software `bazel-remote` supports ZSTD encoding on both HTTP and GRPC protocols, and it can even store ZSTD-compressed casblobs and then serve them directly to HTTP clients.

So `bazel` may also support the `Accept-Encoding: zstd` HTTP requests (besides `gzip`), just like what https://github.com/bazelbuild/bazel/pull/14041 has added into GPRC requests.

### Which category does this issue belong to?

Remote Execution

### What underlying problem are you trying to solve with this feature?

To make HTTP traffic much smaller and also faster.
If a remote-cache server is `bazel-remote`, `Accept-Encoding: zstd` will also save the server's CPU and power.

### Which operating system are you running Bazel on?

Ubuntu 20.04 LTS (in docker)

### What is the output of `bazel info release`?

release 6.5.0; while `master` on 2024-08-29 still lacks such feature

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

N/A

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
Private repos of my work office
```

### Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/13344
https://github.com/bazelbuild/bazel/pull/14041

### Any other information, logs, or outputs that you want to share?

`src/main/java/com/google/devtools/build/lib/remote/GrpcCacheClient.java` has ZSTD-decompression support;
while `src/main/java/com/google/devtools/build/lib/remote/http/HttpCacheClient.java` uses `io.netty.handler.codec.http.HttpContentDecompressor`, which only recognizes `gzip` and `deflate` as said in https://netty.io/4.0/api/io/netty/handler/codec/http/HttpContentDecompressor.html.

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/com/google/devtools/build/lib/remote/http/HttpCacheClient.java and compare its HTTP decompression path with src/main/java/com/google/devtools/build/lib/remote/GrpcCacheClient.java and PR 14041. Check how Netty's HttpContentDecompressor handles encodings, then identify the relevant remote-cache tests. Done means HTTP cache requests can negotiate zstd and correctly consume zstd responses alongside existing encodings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.