Revisit global retry strategy

Meta

1. There a multiple layers where we want to retry failed requests:

• storage
• metastore
• gRPC

2. For requests with no latency constraints, we want to use retry strategies that maximize the chances that one attempt succeeds after a few seconds or even minutes and minimize retry storms. An example of such a case is an indexer uploading a split that was just merged.

3. For requests with latency constraints, we want to use retry strategies that are maybe more aggressive and fail fast (after a few seconds). An example of such a case is a root node sending a search request to a leaf node. Usually, those requests exist to satisfy the demand of a human at the top of the "request tree".

4. For each failure, we need to be able to distinguish transient errors from permanent faults. Otherwise, we can't possibly decide which requests to retry.

Identified problems with the current state of retries in Quickwit

1. We use a single policy at the storage layer to handle both kinds of requests.
2. The metastore layer lacks retry logic (at least for the PostgreSQL implementation).
3. Our return types do not always encode which components are failing and whether the errors are retryable.
4. Part of our current retry logic is untested.

The solution

Racking my brain for a complete proposal but addressing some of the above-mentioned points would be a good start :)

Start by tracing retry handling through the storage, metastore PostgreSQL, and gRPC layers, then inspect the return types and existing retry tests. This issue is complete only after the retry strategy, transient-versus-permanent error handling, and missing test coverage have been defined and agreed.