bazelbuild / bazelbuild/remote-apis

V3 idea: Convert Digest.hash into a oneof for each hashing algorithm

Open
#136 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
445
Forks
141
PR merge metrics
No merged PRs in 30d

Description

Consider the case that you have a storage service that supports multiple hashing algorithms. If someone downloads/uploads a blob, you can currently derive the hashing algorithm that was used by checking the length of Digest.hash. 32 hexadecimal characters? Likely MD5. 64 of them? Probably SHA-256. This allows a storage service to do integrity checking.

This approach becomes problematic if people want to use other hashing algorithms that use the same size. Even worse, modern hashing algorithms like [BLAKE3](https://github.com/BLAKE3-team/BLAKE3/) use a XOF (Extendable Output Function) where the digest length is user configurable. There is no proper way you can even derive the hashing algorithm in that case.

One way would be to simply remove the `hash` field, replacing it by a `oneof`:
```proto
message Digest {
oneof hash {
string sha256 = 1;
string blake3 = 2;
...
}
...
}
```
Unfortunately, not all Protobuf implementations (e.g., [the Go one](https://github.com/golang/protobuf/issues/395)) guarantee that `oneof` fields are serialized in a stable way. Maybe we need to simply use separate fields, emulating a `oneof` kind of construct at a higher level.

Contributor guide

Open the contributing guide

Research direction

Start with the Digest message described in the issue and review how the Go implementation serializes oneof fields. Compare the proposed oneof with separate fields and define how supported algorithms and algorithm identification should work; done requires an agreed API design and compatibility plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.