bazelbuild / bazelbuild/remote-apis

Compression support for inlined data (v2-compatible)

Open
#201 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
445
Forks
141
PR merge metrics
No merged PRs in 30d

Description

We added support for compressed blobs via the bytestream API a while back (#147), and this has been working well so far. Given that we don't seem to be gearing up for a non-backwards compatible REAPIv3 release, I think we should start investigating adding backwards-compatible support for compression of inlined data, aiming to minimize roundtrips while still using compressed data.

One idea that I have been thinking about is to create a new `Blob` message which has the same fields as `Digest`, but with two additional optional fields: `bytes data` to hold the inlined data and `Compressor.Value compressor` to describe the encoding. Then, we would replace each usage of `Digest` in a message that currently also has a separate field for inlined (uncompressed) data with a `Blob` field, and deprecate the previous data field.

For example:
````
message Blob {
// The first two field types + numbers must match those in Digest
string hash = 1;
int64 size_bytes = 2; // This refers to the uncompressed size

reserved 3, 4; // Leave some room in case we want to extend Digest later(?)

bytes data = 5; // Possibly compressed data, if set
Compressor.Value compressor = 6; // Encoding used, if the data field is set
}
````

For "request" messages that can be responded to with inlined data, we would add a repeated field that specifies the compression types that the client accepts. The server would pick one of those encodings (or identity/uncompressed) for each Blob that it chooses to inline data for. We would also add a new field in the Capabilities API for servers to advertise support for this.

I believe this would be backwards-compatible- old clients would not request compressed inlined data, and would receive `Blobs` that are decodable as `Digests` in response, and servers would have a clear signal for when this feature can be used in responses.

Is anyone else interested in this feature?

Contributor guide

Open the contributing guide

Research direction

Review the existing compressed-blob support in the bytestream API from issue #147, then inspect the Digest fields and the request and Capabilities API messages that carry inline data. Decide whether a new Blob message, accepted-compressor fields, and a capability signal provide a backward-compatible design; done means the protocol proposal is resolved and its compatibility is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.