envoyproxy / envoyproxy/gateway
fix(proxy): promote shutdown-manager to native sidecar to close preStop drain race on k8s 1.33+
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 148
Description
On Kubernetes 1.33+, `envoy` and `shutdown-manager` are ordinary sibling containers with no termination ordering guarantee. When kubelet sends SIGTERM during a pod rollout, both containers receive it simultaneously.
Envoy's preStop hook (`httpGet :19002/shutdown/ready`) is designed to block until `shutdown-manager` confirms that in-flight connections have drained. But because `shutdown-manager` can exit before the hook completes its polling loop, the hook fails immediately with `FailedPreStopHook` rather than waiting out the drain window. Kubelet then kills `envoy` with requests still in flight, producing client-visible 5xx errors on every proxy rollout.
**Confirmed on**: Envoy AI Gateway v1.7.0, Kubernetes 1.33
**Reproduce**
1. Deploy a proxy with active traffic
2. Trigger a proxy pod rollout (update `EnvoyProxy` or any setting causing pod recreation)
3. Observe `FailedPreStopHook` events followed by connection-refused on port 19002:
```
Warning FailedPreStopHook pod/envoy- preStop hook for container "envoy" failed
```
Surfaced during a routine gateway rollout — a cluster of 5xx responses traceable to `FailedPreStopHook` events immediately followed by refused connections to the shutdown-manager port.
**Expected behaviour**
`shutdown-manager` should outlive `envoy` so the preStop drain handshake always completes before kubelet kills the proxy process.
**Proposed fix**
On Kubernetes 1.33+, promote `shutdown-manager` from `spec.containers` to `spec.initContainers` with `restartPolicy: Always`. This makes it a [native sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), which kubelet guarantees will keep running until all non-sidecar containers (`envoy`) have fully exited, closing the race entirely.
The change belongs in the proxy pod-spec builder (roughly `internal/infrastructure/kubernetes/proxy/resource.go`) behind a version gate:
```go
if k8sVersion >= 1.33 {
shutdownMgr.RestartPolicy = ptr.To(corev1.ContainerRestartPolicyAlways)
spec.InitContainers = append(spec.InitContainers, shutdownMgr)
// remove from spec.Containers
} else {
spec.Containers = append(spec.Containers, shutdownMgr)
}
```
**Why not make the preStop hook retry-tolerant?**
If `shutdown-manager` has already exited, drain has not completed and Envoy should not be killed. Masking a missing `shutdown-manager` in the hook would paper over the race rather than fix it. The correct invariant is that `shutdown-manager` must be alive for the entire duration of Envoy's preStop window.
**Notes**
- This requires Kubernetes 1.33+ and should be a no-op on older clusters
- It would be useful to export `shutdownManagerContainerName` (and `envoyContainerName`) as package-level constants — downstream consumers currently have to hardcode `"shutdown-manager"` as a bare string with no compile-time contract
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in internal/infrastructure/kubernetes/proxy/resource.go, inspect how the shutdown-manager container is added to the proxy pod and how Kubernetes versions are handled. Implement the version-gated native-sidecar behavior described in the issue, then verify generated pod specs preserve the existing layout before Kubernetes 1.33 and keep shutdown-manager alive through Envoy termination on 1.33+.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100