envoyproxy / envoyproxy/gateway

Gateway status stays Programmed=False even after deployment recovers

Open
#9,432 5 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
During downtime of an envoy deployment (e.g. not enough replicas), the Gateway `Programmed` status flips to `False` with message `Envoy replicas unavailable`. The expected behavior is for that status to flip back to `True` after deployment recovers, however that doesn't happen 100% of the time.

Sometimes the status gets stuck at `False` and the only way to trigger it is via restart of the envoy-gateway or by modifying envoy's deployment (changing annotations is enough).

*Repro steps*:
Hard to consistently reproduce, most likely to happen (in our case) is when control plane is being updated and some change to envoy pods causes deployment to change number of ready replicas. It also seems to happen most often on clusters where we only have one replica of envoy.

*Environment*:
Gateway at commit https://github.com/envoyproxy/gateway/commit/be3969360d01169eb9f6783b4b5689a787eb0183 , was also happening on 1.8.0
Azure AKS and self-hosted Talos Kubernetes cluster

*Logs*:
```
{"level":"info","ts":1783313852.0983703,"logger":"provider","caller":"kubernetes/controller.go:1767","msg":"processing Gateway","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.098473,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"envoy-gateway","name":"http-redirect-private"}
{"level":"info","ts":1783313852.098485,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754a32c7c09fd1","name":"-private"}
{"level":"info","ts":1783313852.0985548,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754d3a0aa8a1f1","name":"-private"}
{"level":"info","ts":1783313852.0986507,"logger":"provider","caller":"kubernetes/controller.go:2985","msg":"processing EnvoyProxy","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.0991585,"logger":"provider","caller":"kubernetes/controller.go:689","msg":"processing Backend","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.099165,"logger":"provider","caller":"kubernetes/controller.go:707","msg":"added Service to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.0991762,"logger":"provider","caller":"kubernetes/controller.go:841","msg":"added EndpointSlice to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private","namespace":"envoy-gateway","name":"envoy-private-dvpjj"}
{"level":"info","ts":1783313852.1246386,"logger":"provider","caller":"kubernetes/controller.go:1767","msg":"processing Gateway","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.1247153,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"envoy-gateway","name":"http-redirect-private"}
{"level":"info","ts":1783313852.1247413,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754a32c7c09fd1","name":"-private"}
{"level":"info","ts":1783313852.1247916,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754d3a0aa8a1f1","name":"-private"}
{"level":"info","ts":1783313852.124897,"logger":"provider","caller":"kubernetes/controller.go:2985","msg":"processing EnvoyProxy","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.1252842,"logger":"provider","caller":"kubernetes/controller.go:689","msg":"processing Backend","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.1252956,"logger":"provider","caller":"kubernetes/controller.go:707","msg":"added Service to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.1253135,"logger":"provider","caller":"kubernetes/controller.go:841","msg":"added EndpointSlice to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private","namespace":"envoy-gateway","name":"envoy-private-dvpjj"}
{"level":"info","ts":1783313852.1625376,"logger":"gateway-api","caller":"runner/runner.go:481","msg":"update envoyproxy status","runner":"gateway-api","key":{"name":"private","namespace":"envoy-gateway"}}
{"level":"info","ts":1783313852.164981,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:29 for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment Resources:map[envoy-gateway/private:{} httproute/argocd/argocd-server/rule/0:{} httproute/argocd/argocd-server/rule/1:{} httproute/linkerd-viz/web/rule/0:{} httproute/-b8754a32c7c09fd1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/1:{} httproute/-b8754d3a0aa8a1f1/-private/rule/2:{} httproute/-b8754d3a0aa8a1f1/-private/rule/3:{} httproute/-b8754d3a0aa8a1f1/-private/rule/4:{} httproute/-b8754d3a0aa8a1f1/-private/rule/5:{} httproute/-b8754d3a0aa8a1f1/-private/rule/6:{}] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"5\""}
{"level":"info","ts":1783313852.1650102,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:30 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map[] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"5\""}
{"level":"info","ts":1783313852.18074,"logger":"provider","caller":"kubernetes/status_updater.go:142","msg":"received a status update","runner":"provider","namespace":"envoy-gateway","name":"private","kind":"Gateway"}
{"level":"info","ts":1783313852.1999645,"logger":"gateway-api","caller":"runner/runner.go:481","msg":"update envoyproxy status","runner":"gateway-api","key":{"name":"private","namespace":"envoy-gateway"}}
{"level":"info","ts":1783313852.2019138,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:31 for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment Resources:map[envoy-gateway/private:{} httproute/argocd/argocd-server/rule/0:{} httproute/argocd/argocd-server/rule/1:{} httproute/linkerd-viz/web/rule/0:{} httproute/-b8754a32c7c09fd1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/1:{} httproute/-b8754d3a0aa8a1f1/-private/rule/2:{} httproute/-b8754d3a0aa8a1f1/-private/rule/3:{} httproute/-b8754d3a0aa8a1f1/-private/rule/4:{} httproute/-b8754d3a0aa8a1f1/-private/rule/5:{} httproute/-b8754d3a0aa8a1f1/-private/rule/6:{}] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"6\""}
{"level":"info","ts":1783313852.20194,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:32 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map[] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"6\""}
{"level":"info","ts":1783313852.2212203,"logger":"provider","caller":"kubernetes/status_updater.go:142","msg":"received a status update","runner":"provider","namespace":"envoy-gateway","name":"private","kind":"Gateway"}
{"level":"info","ts":1783313852.2214108,"logger":"provider","caller":"kubernetes/status_updater.go:107","msg":"status unchanged, bypassing update","runner":"provider","name":"private","namespace":"envoy-gateway","kind":"Gateway"}
{"level":"info","ts":1783313852.3985362,"logger":"infrastructure","caller":"runner/runner.go:110","msg":"received an update","runner":"infrastructure","key":"envoy-gateway/private","delete":false}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with kubernetes/status_updater.go around lines 107-142 and the Gateway processing path in kubernetes/controller.go around line 1767, then trace the status update at runner/runner.go:481. Reproduce recovery after Envoy deployment replicas become unavailable, especially with one replica. Done means Gateway Programmed status reliably returns to True without restarting Envoy Gateway or modifying the deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.