envoyproxy / envoyproxy/gateway
Gateway status stays Programmed=False even after deployment recovers
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
*Description*:
During downtime of an envoy deployment (e.g. not enough replicas), the Gateway `Programmed` status flips to `False` with message `Envoy replicas unavailable`. The expected behavior is for that status to flip back to `True` after deployment recovers, however that doesn't happen 100% of the time.
Sometimes the status gets stuck at `False` and the only way to trigger it is via restart of the envoy-gateway or by modifying envoy's deployment (changing annotations is enough).
*Repro steps*:
Hard to consistently reproduce, most likely to happen (in our case) is when control plane is being updated and some change to envoy pods causes deployment to change number of ready replicas. It also seems to happen most often on clusters where we only have one replica of envoy.
*Environment*:
Gateway at commit https://github.com/envoyproxy/gateway/commit/be3969360d01169eb9f6783b4b5689a787eb0183 , was also happening on 1.8.0
Azure AKS and self-hosted Talos Kubernetes cluster
*Logs*:
```
{"level":"info","ts":1783313852.0983703,"logger":"provider","caller":"kubernetes/controller.go:1767","msg":"processing Gateway","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.098473,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"envoy-gateway","name":"http-redirect-private"}
{"level":"info","ts":1783313852.098485,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754a32c7c09fd1","name":"-private"}
{"level":"info","ts":1783313852.0985548,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754d3a0aa8a1f1","name":"-private"}
{"level":"info","ts":1783313852.0986507,"logger":"provider","caller":"kubernetes/controller.go:2985","msg":"processing EnvoyProxy","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.0991585,"logger":"provider","caller":"kubernetes/controller.go:689","msg":"processing Backend","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.099165,"logger":"provider","caller":"kubernetes/controller.go:707","msg":"added Service to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.0991762,"logger":"provider","caller":"kubernetes/controller.go:841","msg":"added EndpointSlice to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private","namespace":"envoy-gateway","name":"envoy-private-dvpjj"}
{"level":"info","ts":1783313852.1246386,"logger":"provider","caller":"kubernetes/controller.go:1767","msg":"processing Gateway","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.1247153,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"envoy-gateway","name":"http-redirect-private"}
{"level":"info","ts":1783313852.1247413,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754a32c7c09fd1","name":"-private"}
{"level":"info","ts":1783313852.1247916,"logger":"provider","caller":"kubernetes/routes.go:287","msg":"processing HTTPRoute","runner":"provider","namespace":"-b8754d3a0aa8a1f1","name":"-private"}
{"level":"info","ts":1783313852.124897,"logger":"provider","caller":"kubernetes/controller.go:2985","msg":"processing EnvoyProxy","runner":"provider","namespace":"envoy-gateway","name":"private"}
{"level":"info","ts":1783313852.1252842,"logger":"provider","caller":"kubernetes/controller.go:689","msg":"processing Backend","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.1252956,"logger":"provider","caller":"kubernetes/controller.go:707","msg":"added Service to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private"}
{"level":"info","ts":1783313852.1253135,"logger":"provider","caller":"kubernetes/controller.go:841","msg":"added EndpointSlice to resource tree","runner":"provider","kind":"Service","namespace":"envoy-gateway","name":"envoy-private","namespace":"envoy-gateway","name":"envoy-private-dvpjj"}
{"level":"info","ts":1783313852.1625376,"logger":"gateway-api","caller":"runner/runner.go:481","msg":"update envoyproxy status","runner":"gateway-api","key":{"name":"private","namespace":"envoy-gateway"}}
{"level":"info","ts":1783313852.164981,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:29 for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment Resources:map[envoy-gateway/private:{} httproute/argocd/argocd-server/rule/0:{} httproute/argocd/argocd-server/rule/1:{} httproute/linkerd-viz/web/rule/0:{} httproute/-b8754a32c7c09fd1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/1:{} httproute/-b8754d3a0aa8a1f1/-private/rule/2:{} httproute/-b8754d3a0aa8a1f1/-private/rule/3:{} httproute/-b8754d3a0aa8a1f1/-private/rule/4:{} httproute/-b8754d3a0aa8a1f1/-private/rule/5:{} httproute/-b8754d3a0aa8a1f1/-private/rule/6:{}] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"5\""}
{"level":"info","ts":1783313852.1650102,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:30 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map[] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"5\""}
{"level":"info","ts":1783313852.18074,"logger":"provider","caller":"kubernetes/status_updater.go:142","msg":"received a status update","runner":"provider","namespace":"envoy-gateway","name":"private","kind":"Gateway"}
{"level":"info","ts":1783313852.1999645,"logger":"gateway-api","caller":"runner/runner.go:481","msg":"update envoyproxy status","runner":"gateway-api","key":{"name":"private","namespace":"envoy-gateway"}}
{"level":"info","ts":1783313852.2019138,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:31 for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment Resources:map[envoy-gateway/private:{} httproute/argocd/argocd-server/rule/0:{} httproute/argocd/argocd-server/rule/1:{} httproute/linkerd-viz/web/rule/0:{} httproute/-b8754a32c7c09fd1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/0:{} httproute/-b8754d3a0aa8a1f1/-private/rule/1:{} httproute/-b8754d3a0aa8a1f1/-private/rule/2:{} httproute/-b8754d3a0aa8a1f1/-private/rule/3:{} httproute/-b8754d3a0aa8a1f1/-private/rule/4:{} httproute/-b8754d3a0aa8a1f1/-private/rule/5:{} httproute/-b8754d3a0aa8a1f1/-private/rule/6:{}] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"6\""}
{"level":"info","ts":1783313852.20194,"logger":"xds","caller":"v3/simple.go:693","msg":"open delta watch ID:32 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map[] from nodeID: \"envoy-private-6f4c6455f7-b286h\", version \"6\""}
{"level":"info","ts":1783313852.2212203,"logger":"provider","caller":"kubernetes/status_updater.go:142","msg":"received a status update","runner":"provider","namespace":"envoy-gateway","name":"private","kind":"Gateway"}
{"level":"info","ts":1783313852.2214108,"logger":"provider","caller":"kubernetes/status_updater.go:107","msg":"status unchanged, bypassing update","runner":"provider","name":"private","namespace":"envoy-gateway","kind":"Gateway"}
{"level":"info","ts":1783313852.3985362,"logger":"infrastructure","caller":"runner/runner.go:110","msg":"received an update","runner":"infrastructure","key":"envoy-gateway/private","delete":false}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with kubernetes/status_updater.go around lines 107-142 and the Gateway processing path in kubernetes/controller.go around line 1767, then trace the status update at runner/runner.go:481. Reproduce recovery after Envoy deployment replicas become unavailable, especially with one replica. Done means Gateway Programmed status reliably returns to True without restarting Envoy Gateway or modifying the deployment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100