[FeatureRequest] Support ZSTD encoding for HTTP remote cache server
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
The common remote-cache server software `bazel-remote` supports ZSTD encoding on both HTTP and GRPC protocols, and it can even store ZSTD-compressed casblobs and then serve them directly to HTTP clients.
So `bazel` may also support the `Accept-Encoding: zstd` HTTP requests (besides `gzip`), just like what https://github.com/bazelbuild/bazel/pull/14041 has added into GPRC requests.
### Which category does this issue belong to?
Remote Execution
### What underlying problem are you trying to solve with this feature?
To make HTTP traffic much smaller and also faster.
If a remote-cache server is `bazel-remote`, `Accept-Encoding: zstd` will also save the server's CPU and power.
### Which operating system are you running Bazel on?
Ubuntu 20.04 LTS (in docker)
### What is the output of `bazel info release`?
release 6.5.0; while `master` on 2024-08-29 still lacks such feature
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
N/A
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
Private repos of my work office
```
### Have you found anything relevant by searching the web?
https://github.com/bazelbuild/bazel/issues/13344
https://github.com/bazelbuild/bazel/pull/14041
### Any other information, logs, or outputs that you want to share?
`src/main/java/com/google/devtools/build/lib/remote/GrpcCacheClient.java` has ZSTD-decompression support;
while `src/main/java/com/google/devtools/build/lib/remote/http/HttpCacheClient.java` uses `io.netty.handler.codec.http.HttpContentDecompressor`, which only recognizes `gzip` and `deflate` as said in https://netty.io/4.0/api/io/netty/handler/codec/http/HttpContentDecompressor.html.
Contributor guide
Research direction
Start with src/main/java/com/google/devtools/build/lib/remote/http/HttpCacheClient.java and compare its HTTP decompression path with src/main/java/com/google/devtools/build/lib/remote/GrpcCacheClient.java and PR 14041. Check how Netty's HttpContentDecompressor handles encodings, then identify the relevant remote-cache tests. Done means HTTP cache requests can negotiate zstd and correctly consume zstd responses alongside existing encodings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100