envoyproxy / envoyproxy/gateway
Temporary WASM HTTP failures in EnvoyExtensionPolicy do not get reconciled automatically
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
*Description*:
>What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.
Consider the following (stripped) EnvoyExtensionPolicy:
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
name: my-plugin
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
wasm:
- name: wasm-filter
rootID: ""
code:
type: HTTP
http:
url: http://my-provider.my-namespace.svc.cluster.local/plugin.wasm
sha256: 98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
failOpen: true
config:
default_directives: default
directives_map:
default:
- Include @recommended-conf
- Include @crs-setup-conf
- Include @owasp_crs/*.conf
```
When there is a temporary failure in downloading said plugin, the ExtensionPolicy will not get automatically reconciled. Because we use a simple pod to host our plugin, the pod may not be started when the extension policy gets created. The Gateway Controller then tries to download the plugin, but fails.
However, the controller does not retry reconciling said plugin. One has to explicitly trigger the reconciliation. And even annotating the CR does not work. One has to really change the underlying configuration to get the Gateway to do something.
The same happens with OCI images. When there is a temporary failure downloading the OCI image, it will not get retried.
This is contrary to the "Kubernetes Way" where reconciling will be tried again and again until it finally succeeds".
*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.
- Add an Envoy extension policy containing a reference to an unknown service.
- Observe that the Extension Policy's status says that it failed to get configured.
- Apply a simple Pod and Service with an HTTP server hosting a plugin. You can use this Dockerfile to build an image with the Coraza WAF plugin.
```Dockerfile
FROM debian:12 AS builder
RUN apt update && \
apt -y install unzip wget
# renovate: datasource=github-tags depName=corazawaf/coraza-proxy-wasm:
ARG CORAZA_WAF_VERSION="0.5.0"
RUN wget -O coraza-proxy-wasm.zip https://github.com/corazawaf/coraza-proxy-wasm/releases/download/${CORAZA_WAF_VERSION}/coraza-proxy-wasm-${CORAZA_WAF_VERSION}.zip && \
unzip coraza-proxy-wasm.zip && \
mv coraza-proxy-wasm.wasm plugin.wasm && \
sha256sum plugin.wasm
FROM nginxinc/nginx-unprivileged:1.27.3 AS final
COPY --from=builder plugin.wasm /usr/share/nginx/html
```
- Observe that, even after hours, the Envoy Extension Policy will not reconcile and the HTTP server pod does not log any activity at all.
*Environment*:
>Include the environment like gateway version, envoy version and so on.
Gateway 1.3.0 with the default envoy proxy version.
K8s 1.31.2
*Logs*:
>Include the access logs and the Envoy logs.
An example when the SHA checksum is (currently) invalid, but will be valid once the new pods spin up
```yaml
status:
ancestors:
- ancestorRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: my-namespace
conditions:
- lastTransitionTime: "2025-03-27T11:54:29Z"
message: 'Wasm: module downloaded from http://my-provider.my-namespace.svc.cluster.local/plugin.wasm
has checksum ffff6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4,
which does not match: 98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4.'
observedGeneration: 8
reason: Invalid
status: "False"
type: Accepted
controllerName: gateway.envoyproxy.io/gatewayclass-controller
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.