envoyproxy / envoyproxy/gateway

fix(proxy): promote shutdown-manager to native sidecar to close preStop drain race on k8s 1.33+

Open
#9,484 3 comments 3 reactions 0 assignees View on GitHub
stale
Dominant language
Go
Stars
3k
Forks
864
Avg merge
1d 22h
Merged PRs (30d)
148

Description

On Kubernetes 1.33+, `envoy` and `shutdown-manager` are ordinary sibling containers with no termination ordering guarantee. When kubelet sends SIGTERM during a pod rollout, both containers receive it simultaneously.

Envoy's preStop hook (`httpGet :19002/shutdown/ready`) is designed to block until `shutdown-manager` confirms that in-flight connections have drained. But because `shutdown-manager` can exit before the hook completes its polling loop, the hook fails immediately with `FailedPreStopHook` rather than waiting out the drain window. Kubelet then kills `envoy` with requests still in flight, producing client-visible 5xx errors on every proxy rollout.

**Confirmed on**: Envoy AI Gateway v1.7.0, Kubernetes 1.33

**Reproduce**

1. Deploy a proxy with active traffic
2. Trigger a proxy pod rollout (update `EnvoyProxy` or any setting causing pod recreation)
3. Observe `FailedPreStopHook` events followed by connection-refused on port 19002:

```
Warning FailedPreStopHook pod/envoy- preStop hook for container "envoy" failed
```

Surfaced during a routine gateway rollout — a cluster of 5xx responses traceable to `FailedPreStopHook` events immediately followed by refused connections to the shutdown-manager port.

**Expected behaviour**

`shutdown-manager` should outlive `envoy` so the preStop drain handshake always completes before kubelet kills the proxy process.

**Proposed fix**

On Kubernetes 1.33+, promote `shutdown-manager` from `spec.containers` to `spec.initContainers` with `restartPolicy: Always`. This makes it a [native sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), which kubelet guarantees will keep running until all non-sidecar containers (`envoy`) have fully exited, closing the race entirely.

The change belongs in the proxy pod-spec builder (roughly `internal/infrastructure/kubernetes/proxy/resource.go`) behind a version gate:

```go
if k8sVersion >= 1.33 {
shutdownMgr.RestartPolicy = ptr.To(corev1.ContainerRestartPolicyAlways)
spec.InitContainers = append(spec.InitContainers, shutdownMgr)
// remove from spec.Containers
} else {
spec.Containers = append(spec.Containers, shutdownMgr)
}
```

**Why not make the preStop hook retry-tolerant?**

If `shutdown-manager` has already exited, drain has not completed and Envoy should not be killed. Masking a missing `shutdown-manager` in the hook would paper over the race rather than fix it. The correct invariant is that `shutdown-manager` must be alive for the entire duration of Envoy's preStop window.

**Notes**

- This requires Kubernetes 1.33+ and should be a no-op on older clusters
- It would be useful to export `shutdownManagerContainerName` (and `envoyContainerName`) as package-level constants — downstream consumers currently have to hardcode `"shutdown-manager"` as a bare string with no compile-time contract

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/infrastructure/kubernetes/proxy/resource.go, inspect how the shutdown-manager container is added to the proxy pod and how Kubernetes versions are handled. Implement the version-gated native-sidecar behavior described in the issue, then verify generated pod specs preserve the existing layout before Kubernetes 1.33 and keep shutdown-manager alive through Envoy termination on 1.33+.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.