envoyproxy / envoyproxy/gateway

Infrastructure runner does not retry a failed CreateOrUpdateProxyInfra, leaving the Gateway permanently Programmed=False

Open
#10,000 1 comment 0 reactions 0 assignees View on GitHub
area/infra-mgr
Dominant language
Go
Stars
3k
Forks
864
Avg merge
1d 22h
Merged PRs (30d)
148

Description

### Description

When `CreateOrUpdateProxyInfra` fails, the infrastructure runner logs the error and stops. Nothing re-delivers the update, so a **transient** failure leaves the Gateway permanently `Programmed=False` — long after the underlying cause has cleared.

We hit this on EKS with Envoy Gateway **v1.9.0**. A Gateway had no Service and no load balancer for **30 minutes** after the cause had resolved itself in 5. A controller restart fixed it in 15 seconds.

### Observed timeline

| Time (UTC) | Event |
|---|---|
| 19:06:19 | `CreateOrUpdateProxyInfra` fails — the AWS Load Balancer Controller's admission webhook was briefly presenting a certificate the API server did not trust |
| 19:11:02 | That webhook recovers on its own |
| 19:36 | **Still broken.** No further log lines, no retry. Gateway `Accepted=True`, `Programmed=False`, no Service created |
| — | `kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway` → Service created, load balancer assigned, `Programmed=True`, in ~15s |

Controller log, once, and never again:

```
error infrastructure runner/runner.go:144 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update service envoy-gateway-system/envoy-...: Internal error occurred: failed calling webhook \"mservice.elbv2.k8s.aws\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
error infrastructure message/watchutil.go:94 observed an error
```

### Where it comes from

`internal/infrastructure/runner/runner.go` — identical in v1.9.0 and v1.9.1:

```go
if err := r.mgr.CreateOrUpdateProxyInfra(ctx, val); err != nil {
...
r.Logger.Error(err, "failed to create new infra")
errChan <- err
}
```

And `internal/message/watchutil.go` — the only consumer of that channel logs and counts:

```go
errChans := make(chan error, 10)
go func() {
for err := range errChans {
l.Error(err, "observed an error")
watchableSubscribeTotal.WithFailure(metrics.ReasonError, meta.LabelValues()...).Increment()
}
}()
```

So the error is surfaced but never acted upon. Because this is a watch/subscription model rather than reconcile-with-requeue, the runner only tries again when the Infra IR *changes* — and a Gateway that is sitting still produces no further updates.

This looks like the same class of bug v1.9.1 fixed for the extension manager, where the release notes say *"all other errors from the list call are returned so the reconcile is retried instead of publishing an incomplete resource snapshot."*

### Why it is worth fixing rather than working around

Any transient admission-webhook or API-server error during Gateway creation becomes permanent. It fails **silently**: `Accepted=True` suggests the config is fine, and nothing retries or escalates. In our case a GitOps controller also reported the Application `Synced` and `Healthy` throughout, because every manifest it applied did exist — so no tooling flagged it either.

### Suggested fix

Requeue on failure — a bounded retry with backoff in the runner, or re-publishing the IR key so the subscription re-delivers it. Even a slow retry would have closed our 30-minute window automatically.

### Version

- Envoy Gateway **v1.9.0** (`gateway-helm` v1.9.0); code path verified unchanged in **v1.9.1**
- Kubernetes 1.36 (EKS)

Contributor guide

No contributing guide indexed for this repository

Research direction

Read internal/infrastructure/runner/runner.go and internal/message/watchutil.go first, then trace how the infrastructure subscription delivers updates after an error. Verify the chosen retry or re-publication path handles transient CreateOrUpdateProxyInfra failures and eventually creates the Service so the Gateway becomes Programmed=True.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.