Distributed Storage.

Quickwit uses a Storage abstraction to store its data.
While the entire system has been designed to work with an object storage,
there are some benefit to offer a local filesystem storage:
a lower latency, and it is always nice to have that kind of batteries included.

Currently Quickwit offers such a file storage implementation, but this implementation does not work in a distributed environment.
We would like a simple replicated file storage.

Details

Any node should be able to access a file that is on one of the other
node of the cluster through gRPC.

Leaf nodes will host this data.
When distributed a search request amongst the available leaf nodes,
we currently use rendez-vous hashing to define some affinity between
a split and a node.

We want the file placement logic to also pick the 2 replica with the highest affinity. (Let's start with a hardcoded number of replicas)

The metastore API can be extended to deal with the new metadata.

Pitfalls

Raft etc. is not required here. We assume the consistency is dealt with in the metastore API.

The solution needs however to ensure that if a put operation is not entirely successful (e.g. files could not be pushed to two replicas), we never end up with "dangling files" that never get deleted.

Out of scope

• This is not an object storage we are building. We do NOT want striping, or erasure coding.
• We do NOT want to let user modifying files. A put request over an existing file can return an error. The storage trait should give a good idea of how restricted the API is.
• In this ticket, you may discuss how the loss of replica should be handled, but do not implement this.

Read the existing Storage abstraction and metastore API first, then trace how leaf nodes handle files and how rendez-vous hashing assigns splits. Define the replicated placement and failure-cleanup behavior, including gRPC access between nodes; done means successful puts expose both replicas without leaving dangling files after partial failure.