envoyproxy / envoyproxy/gateway

Support per-backendRef locality priority (multi-tier failover) via HTTPRouteFilter

Open
#10,007 5 comments 2 reactions 1 assignee Claimed by @kkk777-7 View on GitHub
kind/feature
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:

Today the only way to influence Envoy's locality priority from Envoy Gateway's API is `Backend.spec.fallback`, which is a `bool`. It lands as `ds.Priority = 1` during route translation and reaches Envoy as `LocalityLbEndpoints.priority`.

That gives us exactly two priority levels, and it is scoped to the `Backend` resource rather than to the backendRef that references it. Three things fall out of that:

1. **No multi-tier failover.** There is no way to express primary -> secondary -> tertiary.
2. **No per-route priority.** The same `Backend` cannot be primary for one route rule and a fallback for another. This is a natural shape for LLM/provider routing (route model A to provider 1 with provider 2 as fallback, and model B the other way around), but it applies to any multi-region or multi-provider setup that reuses backends across routes.
3. **Priority failover cannot be combined with `MergeBackends`.** Cluster deduplication is worth having — smaller xDS, less active health-check traffic — but priority failover only works while a rule's backends share a single cluster. EG gets this right for its own `fallback`, disqualifying such a rule from deduplication. A priority the translator cannot see gets no such protection: the rule's backends are deduplicated into separate weighted clusters, and the failover silently degrades into weight-based distribution.

Because of (2), downstream projects reach past the API. Envoy AI Gateway exposes `AIGatewayRoute.spec.rules[].backendRefs[].priority` and applies it from its extension server, after EG's translation has finished, by rewriting `LocalityLbEndpoints.priority` on the generated clusters in `PostTranslateModify`.

### Describe the solution you'd like
Add a `backendPriority` filter to `HTTPRouteFilter`, referenced from `HTTPBackendRef.filters` via `ExtensionRef`. EG already supports backendRef-level extension filters, This is the only per-backendRef extension point Gateway API gives us, and it is the granularity the use case needs.

#### API

```go
type HTTPRouteFilterSpec struct {
// ... existing fields ...

// BackendPriority sets the Envoy locality priority of the endpoints belonging to the
// backendRef this filter is attached to, enabling priority-based failover between the
// backendRefs of a single HTTPRouteRule.
//
// This filter is only valid when referenced from HTTPBackendRef.filters. Referencing it
// from HTTPRouteRule.filters is rejected: a rule-level filter applies uniformly to every
// backendRef of the rule, and a uniform priority carries no failover meaning.
//
// +optional
BackendPriority *HTTPBackendPriorityFilter `json:"backendPriority,omitempty"`
}

// HTTPBackendPriorityFilter configures Envoy's priority-based locality failover for a single
// backendRef. Envoy sends traffic to the lowest priority value that has enough healthy
// endpoints, and only spills over to the next value as that health degrades.
//
// This overrides the priority derived from the referenced Backend's `fallback` field.
// For additional details, see
// https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/priority
type HTTPBackendPriorityFilter struct {
// Value is the locality priority assigned to this backendRef's endpoints.
// 0 is the highest priority.
//
// Within one HTTPRouteRule, the values used across backendRefs must start at 0 and be
// contiguous. A rule that skips a value is rejected with a ResolvedRefs=False condition
// on the route.
//
// +kubebuilder:validation:Minimum=0
// +kubebuilder:validation:Maximum=127
Value uint32 `json:"value"`
}
```

#### Usage

One `HTTPRouteFilter` per distinct priority value; it is reusable across backendRefs and routes.

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
name: priority-1
spec:
backendPriority:
value: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm
spec:
rules:
- matches:
- headers: [{name: x-model, value: gpt-4}]
backendRefs:
- name: primary-provider # priority 0 (default, no filter)
group: gateway.envoyproxy.io
kind: Backend
- name: secondary-provider
group: gateway.envoyproxy.io
kind: Backend
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: HTTPRouteFilter
name: priority-1
```

[optional *Relevant Links*:]

- https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/priority
- https://github.com/theagentrouter/agent-router/blob/main/api/v1beta1/ai_gateway_route.go#L397

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.