Azure / Azure/azure-functions-host

Host-side drain reliability: drain state lost on host restart, admin API ignores drain mode, admin endpoints return 503 on healthy pods

Open
#11,692 0 comments 0 reactions 1 assignee Claimed by @brettsam View on GitHub
area: host bug needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

## Summary

We've been investigating FlexConsumption pod drain failures at scale. After fixing a platform-side bug where FPS silently swallowed `StopTriggerListeners` failures, we found three host-side gaps that prevent reliable drain on a subset of pods.

## 1. Host does not preserve drain state across restarts

When the host crashes or restarts during drain, the new instance starts clean and re-enables trigger listeners -- picking up new messages as if drain was never requested. The platform correctly retries the drain signal, but the host keeps restarting and undoing it.

**Observed behavior:**
- `DrainMode mode enabled` -> `StopAsync complete` (drain succeeds)
- Host error occurs, host instance is torn down
- New host starts: `Initializing Host. StartupCount=1` (drain state lost)
- New host re-enables triggers, starts accepting new messages
- Platform polls drain status -> host reports `Disabled` (because the new instance was never drained)
- Platform re-sends drain signal, but the cycle repeats

This creates a crash-loop-drain cycle where the pod keeps getting drained and un-drained until `FPSMaxDrainMinutes` (60 min) expires and the pod is force-killed.

**Suggested fix:** On startup, check if the pod is already marked for drain by the platform (e.g., pod status is `MarkedForDrain` or `NonHttpDraining`). If so, don't re-enable trigger listeners.

**Impact:** Internal telemetry shows this pattern accounts for ~53% of remaining DisabledPods post-fix on the affected stamps, concentrated on apps with unstable hosts.

## 2. Admin API invocations don't check IsDrainModeEnabled

`POST /admin/functions/{name}` continues accepting invocations after drain is initiated. Trigger listeners stop, but admin-initiated invocations keep `OutstandingInvocations > 0`, so drain never completes within the configured timeout.

**Suggested fix:** Check `IDrainModeManager.IsDrainModeEnabled` in the invoke path and return 409 or 503 when drain is active.

## 3. Admin endpoints return 503 on pods actively processing traffic

Both `/admin/host/drain` and `/admin/host/drain/status` return 503 on pods that are actively processing customer traffic (confirmed via internal telemetry showing high event volume during the 503 window). This prevents the platform from initiating graceful drain at all.

This may be related to #1 -- during crash-loop restarts, the admin endpoint is briefly unavailable. Could also be a readiness middleware or sidecar health-check gate rejecting the request before it reaches the host.

## Environment

- FlexConsumption (Linux, Legion)
- Host version: 4.1047.100.26071
- Observed across multiple regions and stamps

## Related Issues

- #7103 -- Make Graceful Shutdown timeout configurable or increase default value
- #9822 -- Is there a way to extend the shutdown timeout so drain has enough time to finish?
- #11587 -- Race condition in Resume API causes host to enter zombie state (closed, related mechanism)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.