Implement Graceful Rejection Strategy for Pod Shutdown

Description

Implement graceful rejection strategy for new HTTP requests during pod shutdown to prevent connection drops during Kubernetes rolling updates. The current shutdown handler properly receives SIGTERM signals and immediately sets health status to DOWN, but there's a timing window where new requests may still be accepted while the ALB/load balancer transitions based on health check updates.

Problem Context

During Kubernetes rolling updates, there's a timing gap where:

1. Pod receives SIGTERM and starts shutdown process
2. Load balancer (ALB) continues routing new traffic for 20-75 seconds
3. This causes connection drops and failed requests during the transition

Current State

✅ Already Implemented:

• SIGTERM signal handling with immediate health status transition to DOWN
• Proper shutdown hook registration and execution
• Health check endpoints (/readyz, /livez) correctly reflect shutdown state

❌ Gap: New HTTP requests still accepted during ALB transition period

Proposed Solution

Implement immediate graceful rejection when SIGTERM is received:

HTTP Response Strategy

• Status Code: 503 Service Unavailable
• Headers:
  • Connection: close (prevent connection reuse)
  • Retry-After: 1 (client should retry in 1 second)
• Behavior: Reject NEW requests immediately, allow existing requests to complete

Implementation Approach

1. Shutdown Hook: Register SIGTERM handler that sets isShuttingDown = true
2. Request Filter: Check shutdown state at request entry point
3. Graceful Response: Return 503 with appropriate headers for new requests
4. Connection Management: Close connections cleanly after response

Benefits

• Faster Load Balancer Detection: ALB sees 503s and marks pod unhealthy quicker
• Clean Client Behavior: Clients retry on healthy pods automatically
• Zero Request Drops: Existing requests complete normally
• Coordinated Shutdown: Works with Linkerd proxy and ALB health checks

Infrastructure Coordination

• ALB Health Checks: Optimized to detect failures in ~20s (was 75s)
• Linkerd Proxy: 90s connection draining window
• Kubernetes: 120s termination grace period
• Timeline: 20s rejection period + 70s clean draining period

Acceptance Criteria

• Create HTTP filter that detects shutdown state and rejects new requests with 503 responses
• Add proper headers (Connection: close, Retry-After: 1) to rejection responses
• Ensure existing/in-flight requests complete normally during grace period
• Integrate with existing SIGTERM shutdown handler
• Add logging for rejected requests during shutdown phase
• Verify coordination with ALB health check optimization (~20s detection)
• Test behavior during Kubernetes rolling updates
• Update documentation for complete graceful shutdown process

Technical Implementation Notes

• Hook into existing shutdown handler that already sets health status
• Implement at servlet filter level to intercept requests early in processing pipeline
• Coordinate timing with infrastructure components (ALB, Linkerd, K8s)
• Respect Kubernetes terminationGracePeriodSeconds configuration
• Ensure clean connection closure to prevent client-side issues

Testing Scenarios

• Verify 503 responses with proper headers for new requests during shutdown
• Confirm in-flight requests complete successfully within grace period
• Test ALB health check response time (~20s) to 503 responses
• Validate behavior during Kubernetes rolling updates with zero dropped connections

Start by locating the existing SIGTERM shutdown handler, health endpoints, and servlet request entry point described in the issue. Add rejection for new requests while preserving in-flight requests, then verify 503 responses with Connection and Retry-After headers, logging, and behavior during Kubernetes rolling updates.