envoyproxy / envoyproxy/gateway

Circuit-breaker thresholds cannot be configured on extension-server-generated clusters (EPP ext_proc, InferencePool) capped at the 1024 default

Open
#9,997 2 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

Description

Envoy Gateway currently does not provide a Gateway API resource or policy mechanism to configure circuit-breaker thresholds on clusters generated by extension-server `PostTranslateModify` hooks.

This affects clusters created by the Gateway API Inference Extension and AI Gateway, including:
- `envoy.clusters.endpointpicker___ext_proc` — EPP endpoint-picker `ext_proc` cluster
- `ai-gateway-extproc-uds` — AI Gateway body-processor UDS cluster
- `httproute///rule/` — per-rule HTTPRoute clusters

These clusters receive Envoy's compiled-in circuit-breaker defaults, including `max_requests` and `max_pending_requests` of **1024**, and there is currently no Gateway API resource that can change those values.

For inference gateways, this can impose a throughput ceiling before the backend model servers or GPUs are saturated. In our workload, each inference request makes approximately 15 calls through the EPP `ext_proc` cluster. With `max_requests=1024`, this results in a practical ceiling of approximately **65 concurrent requests**. Requests beyond that point receive 503/500 responses from the Envoy cluster due to circuit-breaker overflow rather than from a configurable policy.

### Why existing mechanisms do not work
**BackendTrafficPolicy cannot target these clusters.**
BackendTrafficPolicy` attaches to Gateway API objects such as Gateway and HTTPRoute, whereas these extension-generated clusters do not have a corresponding Gateway API object that can be targeted.

**EnvoyPatchPolicy is applied before the extension hooks run.**
The extension server's `PostTranslateModify` output runs after the relevant patching stage, so an `EnvoyPatchPolicy` circuit-breaker configuration is subsequently overwritten by the extension-generated output.

### Current workaround
We worked around the limitation by using Envoy Gateway's chained `extensionManagers` to add a second, minimal extension server after the primary extension server.

The second extension server receives the clusters produced by the first extension server and uses `PostTranslateModify` to set `CircuitBreakers` with `DEFAULT` priority on the matching extension-generated clusters. It otherwise returns the clusters, secrets, listeners, and routes unchanged.

Conceptually, the configuration is:
>
> ```yaml
> extensionManagers:
> - name: ai-gateway
> # existing, unchanged
> # ...
>
> - name: valve
> # additional extension server
> failOpen: true
> hooks:
> xdsTranslator:
> post:
> - Translation
> translation:
> cluster:
> includeAll: true
> service:
> fqdn:
> hostname: valve...
> port: 1065
> ```
With `failOpen: true`, if the second extension server is unavailable, the extension chain continues and the clusters fall back to Envoy's default thresholds.

Using this workaround, we increased `max_requests` and `max_pending_requests` to **80,000**. The values were verified in each data-plane proxy's `/clusters` endpoint, and newly created or scaled replicas inherited the configuration automatically because the modification occurs at control-plane translation time rather than per-pod.

### Desired behavior
Envoy Gateway should provide a supported way to configure circuit-breaker thresholds for clusters generated by extension-server translation hooks, including EPP and AI Gateway `ext_proc` clusters.

Ideally, one of the following approaches should be supported:
1. Allow `BackendTrafficPolicy` (or a new policy type) to target extension-generated / EPP `ext_proc` clusters.
2. Have the extension-server framework preserve circuit-breaker settings configured by `BackendTrafficPolicy` and/or `EnvoyPatchPolicy` instead of overwriting them during `PostTranslateModify`.

The goal is to allow operators of inference gateways to raise circuit-breaker limits when the gateway's generated `ext_proc` clusters are otherwise becoming the throughput bottleneck, without requiring a custom extension server solely to modify generated Envoy cluster configuration.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing BackendTrafficPolicy and EnvoyPatchPolicy handling alongside the extension-server PostTranslateModify hook. Inspect how the generated ext_proc and per-rule clusters are produced, then determine how circuit-breaker settings could be preserved or targeted. Done means operators can configure max_requests and max_pending_requests without a custom extension server, with coverage for the listed generated clusters.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.