envoyproxy / envoyproxy/envoy

`retry_back_off.max_interval` is used as per-try timeout for gRPC services (`core`) retry strategy

Open
#38,847 5 comments 2 reactions 0 assignees View on GitHub
area/grpc area/retry bug enhancement help wanted no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

*Title*: Back-off `retry_back_off.max_interval` is used as per-try timeout for gRPC services (`core`) retry strategy

*Description*:
We noticed that when using `ext_authz` filter with `retry_policy` configured for gRPC service, our service requests were timing out rather quickly. Turned out that `retry_back_off.max_interval` is actually used as per-try timeout when `core` retry policy is converted to `route` retry policy.

[Here](https://github.com/envoyproxy/envoy/blob/108dd56175abe1b4059628f6b399a2075e84d557/source/common/grpc/async_client_impl.cc#L39) we convert [`config.core.v3.RetryPolicy` retry policy](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#envoy-v3-api-msg-config-core-v3-retrypolicy) to [`config.route.v3.RetryPolicy` retry policy](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-msg-config-route-v3-retrypolicy) for gRPC service and that function uses `retry_back_off.max_interval` to populate per-try timeout [here](https://github.com/envoyproxy/envoy/blob/108dd56175abe1b4059628f6b399a2075e84d557/source/common/http/utility.cc#L1486-L1487).

I think we should add dedicated per-try timeout configuration to the `core` retry policy and use that instead.

*Repro steps*:
Define an `ext_authz` filter using gRPC service and configure `retry_back_off.max_interval` set to very low value, like 1 or 2 millisecond. Make sure you have `retry_back_off.base_interval` set lower than the `max_interval`.

Make a request through Envoy that hits the filter and observe it time out.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.