envoyproxy / envoyproxy/gateway
Passive healthchecking/retries don't alleviate race condition on Pod deletion
- Lingua principale
- Go
- Stelle
- 3k
- Fork
- 864
- Merge medio
- 2g 2h
- PR unite (30g)
- 140
Descrizione
*Description*:
During moments of Pod churn (especially around Spot terminations in AWS EC2, for example), the EndpointSlice in k8s is updated to remove Pod IPs that have gone away as a result of Pod deletion. There is a race condition (see [this comment](https://github.com/envoyproxy/gateway/issues/4685#issuecomment-2487467421)) between this update and Envoy being updated via xDS to remove the pod IP from the possible backends of routes.
It is not currently possible to work around this race condition with passive healthchecking and `retryOn` policies via a `BackendTrafficPolicy`.
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: evict-deleted-backends
namespace: kube-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
healthCheck:
passive:
baseEjectionTime: 10s
interval: 2s
maxEjectionPercent: 40
consecutive5XxErrors: 1
retry:
numRetries: 2
perRetry:
backOff:
baseInterval: 500ms
maxInterval: 2000ms
timeout: 1000ms
retryOn:
httpStatusCodes:
- 500
triggers:
- connect-failure
- retriable-status-codes
```
See attached this screenshot of a packet capture on our Envoy Pod:

The IPs in question are:
- `10.12.44.215`: client (with a 5s TCP idle timeout configured)
- `10.12.46.156`: Envoy Pod
- `10.12.106.76`: backend target server, configured as a HTTPRoute on our Gateway
You can see the following flow in this screenshot:
- Packets 920 & 928: A successful HTTP flow between client & Envoy
- Packets 994 & 1064: The backend server Pods are deleted and sent TCP RSTs to close the connection between Envoy and the backend server Pods
- Packet 1084: The client sends another HTTP request to Envoy
- Packet 1085: Envoy attempts to create a new TCP connection to the backend (TCP SYN), despite that Pod already going down
- Packets 1841 & 2931: Envoy retransmits the TCP SYN after 0.5s and then 2s
- Packet 3262: client hits idle timeout of 5 seconds and closes the connection to Envoy with a TCP FIN
The HTTPRoute points at a Kubernetes Service for our backend server with 3 replicas, one in each AZ. Theoretically the client need not know which backend servers are available, and should trust Envoy to do the pooling and work around one backend server being down.
*Workaround*:
It is possible to work around this issue by adding a `preStop` lifecycle hook to the target backend server:
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: application
lifecycle:
preStop:
exec:
command:
- /bin/sh
- '-c'
- sleep 25
```
This immediately puts the Pod into `Terminating` state which removes the IP from the EndpointSlice at the start of the preStop hook, but doesn't remove the underlying Pod for the duration of this `sleep 25`. This also gives EG enough time to remove the Pod IP from xDS before it goes away.
*Expectation*:
I would like for EG to fallback to an existing backend endpoint in the case where one of them is going down. In the packet capture we see Envoy retrying the same backend that has already closed its connections and has gone down. We thought that the passive healthchecking and retrying would enable this, but instead it seems the retry applies only to one TCP stream and the passive healthchecking does not solve our issue.
We would like to set config at the Gateway level to bypass this, rather than having to set preStop hooks on every backend server.
*Environment*:
EG: 1.2.4
Envoy: `envoyproxy/envoy:distroless-v1.32.1`
K8s: 1.31
*Logs*:
No logs were emitted during this occurrence.
cc @evilr00t @sam-burrell
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.