bazelbuild / bazelbuild/remote-apis
Compression support for inlined data (v2-compatible)
- Dominant language
- Go
- Stars
- 445
- Forks
- 141
- PR merge metrics
- No merged PRs in 30d
Description
We added support for compressed blobs via the bytestream API a while back (#147), and this has been working well so far. Given that we don't seem to be gearing up for a non-backwards compatible REAPIv3 release, I think we should start investigating adding backwards-compatible support for compression of inlined data, aiming to minimize roundtrips while still using compressed data.
One idea that I have been thinking about is to create a new `Blob` message which has the same fields as `Digest`, but with two additional optional fields: `bytes data` to hold the inlined data and `Compressor.Value compressor` to describe the encoding. Then, we would replace each usage of `Digest` in a message that currently also has a separate field for inlined (uncompressed) data with a `Blob` field, and deprecate the previous data field.
For example:
````
message Blob {
// The first two field types + numbers must match those in Digest
string hash = 1;
int64 size_bytes = 2; // This refers to the uncompressed size
reserved 3, 4; // Leave some room in case we want to extend Digest later(?)
bytes data = 5; // Possibly compressed data, if set
Compressor.Value compressor = 6; // Encoding used, if the data field is set
}
````
For "request" messages that can be responded to with inlined data, we would add a repeated field that specifies the compression types that the client accepts. The server would pick one of those encodings (or identity/uncompressed) for each Blob that it chooses to inline data for. We would also add a new field in the Capabilities API for servers to advertise support for this.
I believe this would be backwards-compatible- old clients would not request compressed inlined data, and would receive `Blobs` that are decodable as `Digests` in response, and servers would have a clear signal for when this feature can be used in responses.
Is anyone else interested in this feature?
Contributor guide
Research direction
Review the existing compressed-blob support in the bytestream API from issue #147, then inspect the Digest fields and the request and Capabilities API messages that carry inline data. Decide whether a new Blob message, accepted-compressor fields, and a capability signal provide a backward-compatible design; done means the protocol proposal is resolved and its compatibility is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100