envoyproxy / envoyproxy/gateway
Expose `ignore_new_hosts_until_first_hc` in BackendTrafficPolicy HealthCheck
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
## Environment
- Envoy Gateway version: v1.7.1
- Kubernetes version: v1.33.0
## Problem
When active health checks are configured via `BackendTrafficPolicy`, newly added endpoints are immediately marked as **healthy by default** and receive traffic before any health check has run. This causes `503 no_healthy_upstream` or `504` timeout errors when the new endpoint is not actually ready to serve requests.
This is Envoy's default behavior — new hosts in a cluster are considered healthy until a health check proves otherwise. Envoy provides a cluster-level setting `common_lb_config.ignore_new_hosts_until_first_hc` to prevent this, but it is not currently exposed through the `BackendTrafficPolicy` API.
## Setup
### Backend (two endpoints for the same model)
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: my-model-backend-cluster-1
namespace: gateway
spec:
endpoints:
- fqdn:
hostname: model-a.cluster-1.example.com
port: 443
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: my-model-backend-cluster-2
namespace: gateway
spec:
endpoints:
- fqdn:
hostname: model-a.cluster-2.example.com
port: 443
```
### HTTPRoute (routes to both backends)
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: my-model
namespace: gateway
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
rules:
- backendRefs:
- group: gateway.envoyproxy.io
kind: Backend
name: my-model-backend-cluster-1
weight: 1
- group: gateway.envoyproxy.io
kind: Backend
name: my-model-backend-cluster-2
weight: 1
```
### BackendTrafficPolicy (with active health checks)
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: my-model
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: my-model
healthCheck:
active:
type: HTTP
http:
path: /healthz
method: GET
expectedStatuses:
- 200
interval: 2s
timeout: 2s
healthyThreshold: 2
unhealthyThreshold: 3
```
## Expected Behavior
When `my-model-backend-cluster-2` is added to the HTTPRoute, it should **not receive traffic** until it passes the configured active health check (`healthyThreshold: 2` successful checks).
## Actual Behavior
`my-model-backend-cluster-2` receives traffic immediately upon being added to the HTTPRoute, before any health check has run. Since the backend is not yet ready, requests routed to it fail with `504` (upstream timeout) or `503` (no healthy upstream).
## Root Cause
Envoy's default behavior marks new endpoints as healthy before any health check runs. The Envoy cluster config `common_lb_config.ignore_new_hosts_until_first_hc` controls this behavior, but it is not exposed in `BackendTrafficPolicy`.
Currently the only workaround is to use `EnvoyPatchPolicy` to inject this setting per cluster:
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
name: ignore-new-hosts-my-model
namespace: gateway
spec:
type: JSONPatch
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
jsonPatches:
- type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
name: "httproute/gateway/my-model/rule/0"
operation:
op: add
path: "/common_lb_config/ignore_new_hosts_until_first_hc"
value: true
- type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
name: "httproute/gateway/my-model/rule/1"
operation:
op: add
path: "/common_lb_config/ignore_new_hosts_until_first_hc"
value: true
```
This workaround does not scale — it requires a separate `EnvoyPatchPolicy` per model/route, and `EnvoyPatchPolicy` does not support wildcard cluster name matching.
## Proposed Solution
Expose `ignore_new_hosts_until_first_hc` as a field under `BackendTrafficPolicy.spec.healthCheck.active`. When active health checks are enabled and this field is set to `true`, Envoy Gateway should set `common_lb_config.ignore_new_hosts_until_first_hc: true` on the generated cluster config.
Example API:
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: my-model
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: my-model
healthCheck:
active:
type: HTTP
http:
path: /healthz
method: GET
expectedStatuses:
- 200
interval: 2s
timeout: 2s
healthyThreshold: 2
unhealthyThreshold: 3
ignoreNewHostsUntilFirstHealthCheck: true # <-- new field
```
This would result in the following xDS cluster config:
```json
{
"common_lb_config": {
"ignore_new_hosts_until_first_hc": true
}
}
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.