corvus-dotnet / corvus-dotnet/Corvus.Extensions.CosmosDb
Review retry logic in the ForEachAsync extensions for Container
- Dominant language
- C#
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The `CosmosClient` has built-in retry logic. The various `ForEachAsync` extension methods that `CosmosClientExtensions` defines for `Container` wrap calls to the client in another layer of retries: a `Backoff` strategy configured for 3 attempts and a delta of 1 second.
It seems possible that the effective behaviour is not what was intended, because this backoff policy is applied as a layer above the `CosmosClient`'s retry.
So the only situation in which the `ForEachAsync` backoff will do anything is if the built-in retry has already failed. And the built-in policy has two elements:
* throttling-based retries
* endpoint failure-based retries
The throttling defaults to 9 retries in the face of a 429 (request rate exceeded) response, with a maximum of 9 retries or 30 seconds, whichever comes sooner. (This is configurable.)
The endpoint failure retry will attempt to determine whether it can connect to an alternate endpoint and try that instead.
So all this will have had a go before we start our one-second-at-a-time backoff.
That said, it does seem to be looking for a different failure. The retry performed by `ForEachAsync` only seems to occur on a 503 service unavailable. So it's possible that it has been found that sometimes we get random failures of this kind, and that the CosmosClient does not retry in the face of these.
But it's not entirely clear. Moreover, if this is genuinely important, we should be making this available in a more general way, and not just to code that wants to execute a query that processes a range of documents.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.