envoyproxy / envoyproxy/gateway

Gateway stuck at Programmed=False / AddressNotAssigned on v1.8.1 after transient LoadBalancer IP loss; does not self-recover

Open
#9,567 6 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

**TL;DR**

On v1.8.1, when several LoadBalancer-backed Gateways transiently lose their Service `status.loadBalancer.ingress` at the same time (a cloud LB controller resync), the affected Gateways go `Programmed=False` / `AddressNotAssigned` and **stay stuck indefinitely** even though the backing Services regain valid IPs and the data plane keeps routing. The controller does not self-recover; only a controller restart, or forcing a Service `MODIFIED` watch event, brings the Gateway back to `Programmed=True`. This reproduces on a build that already contains the fixes for #8953 (#8962) and #9094 (#9100), so neither of those covers this case.

**Environment**
- Envoy Gateway: **v1.8.1** (contains #8962 and #9100)
- Kubernetes: v1.34, multi-node (managed cloud)
- Load balancer: cloud-provider L4 passthrough (external NLB + internal ILB)
- Gateway API: v1
- Topology: multiple GatewayClasses, each served by its own controller Deployment in a dedicated namespace. The generated envoy `LoadBalancer` Services live in the **controller** namespace; the `Gateway` objects live in application namespaces.

**What happened**
- A cloud LB controller resync briefly cleared `status.loadBalancer.ingress` on several envoy Services at once.
- Every Gateway whose backing Service was affected flipped to `Programmed=False` / `AddressNotAssigned` at the **same second** (5 Gateways across 2 controller namespaces).
- The Services regained their IPs shortly after and kept them; traffic was never interrupted.
- The Gateways stayed stuck for ~8h. During that window the leader controller logged **no reconciles** (idle), and no errors/warnings.
- `kubectl rollout restart deploy/envoy-gateway` in the affected controller namespaces recovered every Gateway to `Programmed=True` with the correct address within seconds.

**Observed while stuck**
```
Gateway: Programmed=False, reason=AddressNotAssigned,
message="No addresses have been assigned to the Gateway", ADDRESS empty
Service: status.loadBalancer.ingress = [{ip: , ipMode: VIP}] # valid the entire time
```

**Recovery — both confirmed on affected clusters (v1.8.1)**
1. `kubectl rollout restart deploy/envoy-gateway -n ` → all Gateways back to `Programmed=True` within seconds.
2. Touching the backing Service to force a `MODIFIED` event recovers a single stuck Gateway in ~5s:
```
kubectl annotate svc envoy-- -n touch="$(date -u +%FT%TZ)" --overwrite
# Gateway -> Programmed=True, correct address, ~5s later
```
This shows the controller is otherwise healthy and the Service IP is valid — the Gateway status is simply not re-derived until a Service watch event fires.

**Why this looks distinct from #8953 / #9094 (both present in v1.8.1)**
- #8953 (#8962): fixed the `MergeConditions` / `LastTransitionTime` `reflect.DeepEqual` re-write; present in v1.8.1 (`conditionChanged` uses `cmp.Equal` ignoring `LastTransitionTime`). Here the controller was *idle* after the flip — it was not re-writing `False`; it simply never re-derived `True`.
- #9094 (#9100): "always include the controller namespace in the watched namespaces"; present in v1.8.1. The controller does watch/serve those namespaces — Gateways/HTTPRoutes in them get status normally.

**Suspected area**
Recovery to `Programmed=True` appears to depend on a subsequent Service `MODIFIED` event re-triggering the infra/address reconcile. If that event is missed (leader-election / cache-warmup window, or informer state after concurrent multi-Service updates), the Gateway stays latched at `AddressNotAssigned` with no periodic resync to self-heal. Is there a path where the Gateway address status is only recomputed on a Service watch event, with no fallback resync?

**Impact**
The data plane is unaffected, but `Gateway.status.addresses` is empty, so external-dns with `--policy=sync` deletes the DNS records for every attached HTTPRoute — a DNS-outage risk despite healthy traffic (same impact called out in #8953).

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the Gateway address/status reconciliation and the watch path from backing Service MODIFIED events. Reproduce the transient loss and recovery scenario with multiple Gateways, then verify that valid Service ingress addresses cause every affected Gateway to return to Programmed=True without a controller restart or manual Service event.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend-api-design, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.