grpc / grpc/grpc-dotnet

Minimum timeout for retry policy

Open
#2,673 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
4.5k
Forks
836
Avg merge
6d 3h
Merged PRs (30d)
7

Description

### Is your feature request related to a problem? Please describe.

Our test environment fails to get going, we think it's because sometimes all the retries happen in quick succession. We have various scripts that orchestrate multiple server components. There are dependencies between these components and sometimes it takes a few seconds for them to all come online, sometimes longer because reasons.

### Describe the solution you'd like

I'd like to extend the retry policy with a minimum delay between retries.

### Describe alternatives you've considered

Write my own gRPC stack...

I looked into the [retry code](https://github.com/grpc/grpc-dotnet/blob/492158d5cbd2c83d8d546849f88701f368058d5e/src/Grpc.Net.Client/Internal/Retry/RetryCall.cs#L209-L217), and there's nothing I can do from my end.

### Additional context

It might have been a mistake to not provide a minimum delay between retries because now there's no way to ensure that at least X seconds is spent waiting for services to come online. The way this is described [elsewhere](https://github.com/grpc/proposal/blob/master/A6-client-retries.md#exponential-backoff) makes me think this might have been an oversight?

I ran some experiments with the default retry policy documented [here](https://learn.microsoft.com/en-us/aspnet/core/grpc/retries?view=aspnetcore-9.0#configure-a-grpc-retry-policy).

Basically, there's about 1/100 chance that all retries happen within 1 second (because there's no minimum).

I think there needs to be a minimum, it can default to zero, but at least this way we can ensure that there's a minimum period where it's waiting before giving up.

What we believe is happening in our test environment is that sometimes it takes a bit longer for various server components to go online which leads to intermittent startup failures which is causing CI/CD to fail more or less randomly.

Contributor guide

Open the contributing guide

Research direction

Start with src/Grpc.Net.Client/Internal/Retry/RetryCall.cs around lines 209-217, then compare its behavior with the A6 client-retries exponential-backoff description. Trace how the documented default retry policy is configured and identify the policy surface and tests that would need to cover a configurable minimum delay, including the default-zero behavior. Done means the requested delay is supported and its retry timing is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, grpc
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.