envoyproxy / envoyproxy/gateway
HTTP/3 + proxyProtocol ClientTrafficPolicy generates invalid QUIC listener and causes continuous LDS NACKs
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
*Description*:
When `http3` and `proxyProtocol` are enabled together on the same `ClientTrafficPolicy`, Envoy Gateway generates a QUIC listener containing the PROXY protocol listener filter.
Envoy rejects the generated LDS update with:
`Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'`
This results in continuous LDS NACKs from every Envoy proxy.
The affected listener in this case is: `envoy-gateway-system/korlark/https-api-quic`
TCP HTTPS traffic continues working because Envoy retains the last valid LDS configuration, so the problem can be easy to miss from Gateway/HTTPRoute status and normal HTTP/2 traffic alone.
In my deployment, both Envoy replicas continuously NACK the same Listener resource. `xds_nack_total` reached 164 for each replica, with 39 additional NACKs per replica during the most recent one-hour window.
Expected behavior:
Envoy Gateway should not generate a QUIC listener configuration that its supported Envoy version rejects.
Possible expected behaviors would be:
1. Reject this `ClientTrafficPolicy` combination during validation with a clear status/error.
2. Disable HTTP/3 for the affected listener and report the reason, similar to the existing handling for unsupported HTTP/3 + downstream client TLS validation combinations.
3. Generate a valid QUIC listener without the incompatible listener filter, if that is semantically supported.
The current behavior allows an apparently valid policy to reach xDS generation and then repeatedly fail at the Envoy dataplane.
*Repro steps*:
Apply a Gateway with an HTTPS listener and the following
`ClientTrafficPolicy`:
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: proxy-protocol-http3
spec:
http3: {}
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: example
proxyProtocol:
optional: true
```
The underlying LoadBalancer in my environment uses PROXY protocol for the TCP data path.
After reconciliation, inspect the Envoy Gateway controller logs:
```bash
kubectl logs -n envoy-gateway-system deploy/envoy-gateway \
--since=1h | grep -Ei 'nack|error_detail|reject|listener'
```
Also inspect:
```promql
sum by (nodeID, typeURL) (
increase(xds_nack_total[1h])
)
```
In my deployment, both Envoy replicas continuously NACK the generated QUIC Listener.
HTTP/1.1 and HTTP/2 traffic through the TCP listener remain functional.
*Environment*:
- Envoy Gateway: v1.9.0
- Envoy Proxy: v1.39.0 (`distroless-v1.39.0`)
- Gateway API: v1.6.1
- Kubernetes: v1.36.1
- Platform: Vultr Kubernetes Engine (VKE)
- Envoy Gateway controller replicas: 1
- Envoy Proxy replicas: 2
- LoadBalancer: Vultr Load Balancer with PROXY protocol enabled
Relevant policy:
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: korlark-proxy-protocol
spec:
http3: {}
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: korlark
proxyProtocol:
optional: true
```
*Logs*:
Both Envoy replicas report the same error:
```text
error xds cache/snapshotcache.go:396 Envoy rejected the last update for type type.googleapis.com/envoy.config.listener.v3.Listener on node envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-zv55k with code 13 and message Error adding/updating listener(s) envoy-gateway-system/korlark/https-api-quic: Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'
error xds cache/snapshotcache.go:396 Envoy rejected the last update for type type.googleapis.com/envoy.config.listener.v3.Listener on node envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-p575h with code 13 and message Error adding/updating listener(s) envoy-gateway-system/korlark/https-api-quic: Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'
```
Observed metrics:
```text
increase(xds_nack_total[1h])
{nodeID="envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-p575h",
typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 39
{nodeID="envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-zv55k",
typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 39
```
Cumulative values:
```text
xds_nack_total{nodeID="...-p575h",
typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 164
xds_nack_total{nodeID="...-zv55k",
typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 164
```
The NACKs are deterministic and occur identically on both dataplane replicas.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.