envoyproxy / envoyproxy/gateway
Infrastructure runner does not retry a failed CreateOrUpdateProxyInfra, leaving the Gateway permanently Programmed=False
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 148
Description
### Description
When `CreateOrUpdateProxyInfra` fails, the infrastructure runner logs the error and stops. Nothing re-delivers the update, so a **transient** failure leaves the Gateway permanently `Programmed=False` — long after the underlying cause has cleared.
We hit this on EKS with Envoy Gateway **v1.9.0**. A Gateway had no Service and no load balancer for **30 minutes** after the cause had resolved itself in 5. A controller restart fixed it in 15 seconds.
### Observed timeline
| Time (UTC) | Event |
|---|---|
| 19:06:19 | `CreateOrUpdateProxyInfra` fails — the AWS Load Balancer Controller's admission webhook was briefly presenting a certificate the API server did not trust |
| 19:11:02 | That webhook recovers on its own |
| 19:36 | **Still broken.** No further log lines, no retry. Gateway `Accepted=True`, `Programmed=False`, no Service created |
| — | `kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway` → Service created, load balancer assigned, `Programmed=True`, in ~15s |
Controller log, once, and never again:
```
error infrastructure runner/runner.go:144 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update service envoy-gateway-system/envoy-...: Internal error occurred: failed calling webhook \"mservice.elbv2.k8s.aws\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
error infrastructure message/watchutil.go:94 observed an error
```
### Where it comes from
`internal/infrastructure/runner/runner.go` — identical in v1.9.0 and v1.9.1:
```go
if err := r.mgr.CreateOrUpdateProxyInfra(ctx, val); err != nil {
...
r.Logger.Error(err, "failed to create new infra")
errChan <- err
}
```
And `internal/message/watchutil.go` — the only consumer of that channel logs and counts:
```go
errChans := make(chan error, 10)
go func() {
for err := range errChans {
l.Error(err, "observed an error")
watchableSubscribeTotal.WithFailure(metrics.ReasonError, meta.LabelValues()...).Increment()
}
}()
```
So the error is surfaced but never acted upon. Because this is a watch/subscription model rather than reconcile-with-requeue, the runner only tries again when the Infra IR *changes* — and a Gateway that is sitting still produces no further updates.
This looks like the same class of bug v1.9.1 fixed for the extension manager, where the release notes say *"all other errors from the list call are returned so the reconcile is retried instead of publishing an incomplete resource snapshot."*
### Why it is worth fixing rather than working around
Any transient admission-webhook or API-server error during Gateway creation becomes permanent. It fails **silently**: `Accepted=True` suggests the config is fine, and nothing retries or escalates. In our case a GitOps controller also reported the Application `Synced` and `Healthy` throughout, because every manifest it applied did exist — so no tooling flagged it either.
### Suggested fix
Requeue on failure — a bounded retry with backoff in the runner, or re-publishing the IR key so the subscription re-delivers it. Even a slow retry would have closed our 30-minute window automatically.
### Version
- Envoy Gateway **v1.9.0** (`gateway-helm` v1.9.0); code path verified unchanged in **v1.9.1**
- Kubernetes 1.36 (EKS)
Contributor guide
No contributing guide indexed for this repository
Research direction
Read internal/infrastructure/runner/runner.go and internal/message/watchutil.go first, then trace how the infrastructure subscription delivers updates after an error. Verify the chosen retry or re-publication path handles transient CreateOrUpdateProxyInfra failures and eventually creates the Service so the Gateway becomes Programmed=True.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100