aws / aws/amazon-eks-pod-identity-webhook
Add preStop hook to prevent webhook failures during pod termination
- Dominant language
- Go
- Stars
- 691
- Forks
- 203
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What would you like to be added**:
Add a preStop hook with sleep action to the webhook deployment to prevent connection failures during pod termination:
```yaml
lifecycle:
preStop:
sleep:
seconds: 10
```
**Why is this needed**:
During Kubernetes cluster operations (node drain, cluster upgrade, pod deletion), the pod-identity-webhook experiences connection failures because pods receive traffic while terminating. This occurs due to the timing gap between pod termination start and Endpoints removal.
**Error examples from kube-apiserver logs:**
```
failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.pod-identity-webhook.svc:443/mutate?timeout=10s": dial tcp 10.100.xxx.xxx:443: connect: connection refused
failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.pod-identity-webhook.svc:443/mutate?timeout=10s": context deadline exceeded
```
**Root cause:**
When a pod terminates, the Endpoints removal happens asynchronously and may take several seconds. During this window, new requests can still be routed to the terminating pod, causing connection failures.
The preStop hook delays SIGTERM delivery, allowing time for:
- Endpoints removal to propagate
- In-flight requests to complete
- kube-proxy iptables rules to update
**Why preStop instead of fixing graceful shutdown:**
While issue #271 addresses application-level graceful shutdown bugs, a preStop hook is simpler and more effective because:
- It works at the Kubernetes level, independent of application code
- It addresses the Endpoints removal timing issue directly
- Native sleep action available since Kubernetes 1.29 (alpha), 1.30+ (beta, enabled by default)
**Related issues:**
- #271 - Application graceful shutdown bug
- #111 - readinessProbe request (addresses startup phase)
- PR #287 - Adds readinessProbe
The preStop hook addresses the termination phase, while readinessProbe addresses the startup phase.
Contributor guide
Research direction
Start by locating the pod-identity-webhook Deployment manifest and reviewing how its configuration is packaged. Add the specified 10-second preStop sleep hook, then verify that the resulting Deployment contains the lifecycle configuration and that the webhook can tolerate pod termination without connection failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100