[Discussion] Make CAS blobs tied to ActionKey to improve sharding locality

Hi there! Opening a discussion issue to see what other folks think about this.

Right now CAS keyspace is basically hashes of contents. This means that if your CAS is a sharded system, actions can have their outputs split among multiple shards. This also means that if you lose just one of your shards, you will force * a lot * of actions to rebuild, as a lot of them lost maybe one or two output files that happened to be on that shard. It also means that generally speaking, latency of most FindMissingBlobs calls is going to depend on tail latency over multiple shards, which is undesirable.

My initial gut feeling is that if we could have some keyeing by a combination of an action key + output key instead of having raw output hashes, then we could use that to have better shard locality ( colocate all action outputs in single shard ). It comes with some rough edges ( for example, it might be that multiple actions have the same output - and we would see duplication ), but in practice I don't think it would be that much of a concern.

Start by reviewing the CAS keyspace and FindMissingBlobs behavior described in the issue, then trace the remote-apis implementation that handles action outputs and sharding. The issue names no files or tests; done would require an agreed design for ActionKey-based locality and its duplication and recovery tradeoffs.

Written by the indexing model from the issue text.