envoyproxy / envoyproxy/gateway

Expose `ignore_new_hosts_until_first_hc` in BackendTrafficPolicy HealthCheck

Open
#8,597 3 comments 1 reaction 1 assignee Claimed by @Aditya7880900936 View on GitHub
kind/feature stale
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

## Environment

- Envoy Gateway version: v1.7.1
- Kubernetes version: v1.33.0

## Problem

When active health checks are configured via `BackendTrafficPolicy`, newly added endpoints are immediately marked as **healthy by default** and receive traffic before any health check has run. This causes `503 no_healthy_upstream` or `504` timeout errors when the new endpoint is not actually ready to serve requests.

This is Envoy's default behavior — new hosts in a cluster are considered healthy until a health check proves otherwise. Envoy provides a cluster-level setting `common_lb_config.ignore_new_hosts_until_first_hc` to prevent this, but it is not currently exposed through the `BackendTrafficPolicy` API.

## Setup

### Backend (two endpoints for the same model)

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: my-model-backend-cluster-1
namespace: gateway
spec:
endpoints:
- fqdn:
hostname: model-a.cluster-1.example.com
port: 443
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: my-model-backend-cluster-2
namespace: gateway
spec:
endpoints:
- fqdn:
hostname: model-a.cluster-2.example.com
port: 443
```

### HTTPRoute (routes to both backends)

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: my-model
namespace: gateway
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
rules:
- backendRefs:
- group: gateway.envoyproxy.io
kind: Backend
name: my-model-backend-cluster-1
weight: 1
- group: gateway.envoyproxy.io
kind: Backend
name: my-model-backend-cluster-2
weight: 1
```

### BackendTrafficPolicy (with active health checks)

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: my-model
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: my-model
healthCheck:
active:
type: HTTP
http:
path: /healthz
method: GET
expectedStatuses:
- 200
interval: 2s
timeout: 2s
healthyThreshold: 2
unhealthyThreshold: 3
```

## Expected Behavior

When `my-model-backend-cluster-2` is added to the HTTPRoute, it should **not receive traffic** until it passes the configured active health check (`healthyThreshold: 2` successful checks).

## Actual Behavior

`my-model-backend-cluster-2` receives traffic immediately upon being added to the HTTPRoute, before any health check has run. Since the backend is not yet ready, requests routed to it fail with `504` (upstream timeout) or `503` (no healthy upstream).

## Root Cause

Envoy's default behavior marks new endpoints as healthy before any health check runs. The Envoy cluster config `common_lb_config.ignore_new_hosts_until_first_hc` controls this behavior, but it is not exposed in `BackendTrafficPolicy`.

Currently the only workaround is to use `EnvoyPatchPolicy` to inject this setting per cluster:

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
name: ignore-new-hosts-my-model
namespace: gateway
spec:
type: JSONPatch
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
jsonPatches:
- type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
name: "httproute/gateway/my-model/rule/0"
operation:
op: add
path: "/common_lb_config/ignore_new_hosts_until_first_hc"
value: true
- type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
name: "httproute/gateway/my-model/rule/1"
operation:
op: add
path: "/common_lb_config/ignore_new_hosts_until_first_hc"
value: true
```

This workaround does not scale — it requires a separate `EnvoyPatchPolicy` per model/route, and `EnvoyPatchPolicy` does not support wildcard cluster name matching.

## Proposed Solution

Expose `ignore_new_hosts_until_first_hc` as a field under `BackendTrafficPolicy.spec.healthCheck.active`. When active health checks are enabled and this field is set to `true`, Envoy Gateway should set `common_lb_config.ignore_new_hosts_until_first_hc: true` on the generated cluster config.

Example API:

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: my-model
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: my-model
healthCheck:
active:
type: HTTP
http:
path: /healthz
method: GET
expectedStatuses:
- 200
interval: 2s
timeout: 2s
healthyThreshold: 2
unhealthyThreshold: 3
ignoreNewHostsUntilFirstHealthCheck: true # <-- new field
```

This would result in the following xDS cluster config:

```json
{
"common_lb_config": {
"ignore_new_hosts_until_first_hc": true
}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.