envoyproxy / envoyproxy/gateway
Data-plane NACKs are not propagated to Gateway/EnvoyProxy status, leaving stale config silently accepted
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
*Description*:
This issue is similar to this one: https://github.com/envoyproxy/gateway/issues/8847
When updating an already deployed EnvoyProxy resources with a faulty config, the controller (i.e envoy-gateway-* pod) silently refuses the update. This means that when new envoy proxy pods will be created (after a node roll, a scaling event...) they will try to load the bad listener and break as they won't have the last good config cached.
This problem is most likely coming from there: https://github.com/envoyproxy/gateway/blob/main/internal/xds/translator/accesslog.go#L177-L186
The JSON format is copied into a stuct and wrapped in a SubstitutionFormatString_Json format but there isn't any check concerning the content.
According to Claude:
- The xDS validation that does exist can't catch it. translator.go calls tCtx.ValidateAll() (the "validation failed for xds resource" path), but that's protoc-gen-validate (PGV) schema validation: it only checks proto field constraints.
- The data-plane NACK is never propagated back to status. Envoy Gateway does not subscribe the ADS NACK / error_detail back onto the Gateway / EnvoyProxy conditions, so the operator gets no feedback.
*Repro steps*:
In a kubernetes cluster:
- Deploy Envoy Gateway with the helm chart.
- Deploy a simple EnvoyProxy resource, a GatewayClass and Gateway using this class and referencing the EnvoyProxy in its `infrastructure` field.
- Make sure the proxy pods generated from the EnvoyProxy resource are running normally and are not logging any error.
- Update the helm chart to introduce a faulty telemetry field in the EnvoyProxy config such as this one:
```yaml
telemetry:
accessLog:
settings:
- format:
json:
http.host: '%REQ(:AUTHORITY)%'
http.method: '%REQ(:METHOD)%'
http.path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
http.protocol: '%PROTOCOL%'
http.referer: '%REQ(REFERER)%'
upstream_status_code: "%UPSTREAM_STATUS%"
type: JSON
matches:
- request.path != "/auth/callback" && request.path != "/healthz"
sinks:
- file:
path: /dev/stdout
type: File
```
- Look at the proxy pods' and controller logs.
The faulty part of the `telemetry` field here is the `"%UPSTREAM_STATUS%"` value which isn't supported.
So in the case of my above protocol here is what I think happened:
- After the chart being edited, the giantswarm-default EnvoyProxy went to generation: 2
- The envoy-gateway-* pod (the controller) reconciled it, translated the new telemetry.accesLog into an updated Listener xDS resource and pushed it over the gRPC ADS stream to the already running proxy pods.
- These received the new Listener but rejected it when trying to instantiate the access-log formatter. They then kept serving the last known good config.
*Environment*:
Gateway version: 1.8.1
*Logs*:
You should see the following log line in the proxy pods:
```
ERROR xds cache/snapshotcache.go:388 Envoy rejected the last update with code 13 and message Error adding/updating listener(s) envoy-gateway-system/internal http: Not supported field in StreamInfo: UPSTREAM_STATUS
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.