aws / aws/amazon-eks-pod-identity-webhook

Add preStop hook to prevent webhook failures during pod termination

Open
#289 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
203
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What would you like to be added**:

Add a preStop hook with sleep action to the webhook deployment to prevent connection failures during pod termination:

```yaml
lifecycle:
preStop:
sleep:
seconds: 10
```

**Why is this needed**:

During Kubernetes cluster operations (node drain, cluster upgrade, pod deletion), the pod-identity-webhook experiences connection failures because pods receive traffic while terminating. This occurs due to the timing gap between pod termination start and Endpoints removal.

**Error examples from kube-apiserver logs:**
```
failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.pod-identity-webhook.svc:443/mutate?timeout=10s": dial tcp 10.100.xxx.xxx:443: connect: connection refused
failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.pod-identity-webhook.svc:443/mutate?timeout=10s": context deadline exceeded
```

**Root cause:**

When a pod terminates, the Endpoints removal happens asynchronously and may take several seconds. During this window, new requests can still be routed to the terminating pod, causing connection failures.

The preStop hook delays SIGTERM delivery, allowing time for:
- Endpoints removal to propagate
- In-flight requests to complete
- kube-proxy iptables rules to update

**Why preStop instead of fixing graceful shutdown:**

While issue #271 addresses application-level graceful shutdown bugs, a preStop hook is simpler and more effective because:
- It works at the Kubernetes level, independent of application code
- It addresses the Endpoints removal timing issue directly
- Native sleep action available since Kubernetes 1.29 (alpha), 1.30+ (beta, enabled by default)

**Related issues:**
- #271 - Application graceful shutdown bug
- #111 - readinessProbe request (addresses startup phase)
- PR #287 - Adds readinessProbe

The preStop hook addresses the termination phase, while readinessProbe addresses the startup phase.

Contributor guide

Open the contributing guide

Research direction

Start by locating the pod-identity-webhook Deployment manifest and reviewing how its configuration is packaged. Add the specified 10-second preStop sleep hook, then verify that the resulting Deployment contains the lifecycle configuration and that the webhook can tolerate pod termination without connection failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.