envoyproxy / envoyproxy/envoy

`rate_limited_retry_back_off` does not sufficiently respect `timeout`

Open
#24,316 5 comments 0 reactions 0 assignees View on GitHub
area/http area/retry bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

*Title*: `rate_limited_retry_back_off` does not sufficiently respect `timeout`

*Description*:
I have set up `rate_limited_retry_back_off` with the header `retry-after`; this works as expected.

However, I have cases where the value `retry-after` is larger than the remaining timeout budget, or the value of `timeout` even. This is the case when we have a server side timeout due to an expensive operation, and do not want any immediate retries.

In these cases, Envoy will try to wait for the amount of time specified by the `retry-after` header, but will eventually time out and respond with a 504.

Since the wait time specified by `retry-after` is _larger_ than the remaining time budget for the request, there is no way the request can succeed; timeout is a certainty. I believe the correct behavior would be to **instantly fail the request**, either with the upstream response or with a 504.

*Repro steps*:

You'll need a service that can generate e.g. a 503 along with a `Retry-After` header with a value _higher_ than 10, ensuring the timeout is enforced.

*Config*:
```yaml
timeout: 10s
retry_policy:
rate_limited_retry_back_off:
max_interval: 300s
reset_headers:
- format: SECONDS
name: Retry-After
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.