corvus-dotnet / corvus-dotnet/Corvus.Extensions.CosmosDb

Review retry logic in the ForEachAsync extensions for Container

Open
#287 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

The `CosmosClient` has built-in retry logic. The various `ForEachAsync` extension methods that `CosmosClientExtensions` defines for `Container` wrap calls to the client in another layer of retries: a `Backoff` strategy configured for 3 attempts and a delta of 1 second.

It seems possible that the effective behaviour is not what was intended, because this backoff policy is applied as a layer above the `CosmosClient`'s retry.

So the only situation in which the `ForEachAsync` backoff will do anything is if the built-in retry has already failed. And the built-in policy has two elements:

* throttling-based retries
* endpoint failure-based retries

The throttling defaults to 9 retries in the face of a 429 (request rate exceeded) response, with a maximum of 9 retries or 30 seconds, whichever comes sooner. (This is configurable.)

The endpoint failure retry will attempt to determine whether it can connect to an alternate endpoint and try that instead.

So all this will have had a go before we start our one-second-at-a-time backoff.

That said, it does seem to be looking for a different failure. The retry performed by `ForEachAsync` only seems to occur on a 503 service unavailable. So it's possible that it has been found that sometimes we get random failures of this kind, and that the CosmosClient does not retry in the face of these.

But it's not entirely clear. Moreover, if this is genuinely important, we should be making this available in a more general way, and not just to code that wants to execute a query that processes a range of documents.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.