Azure / Azure/azure-functions-host

Host deadlock after worker startup timeout

Open
#11,617 2 comments 0 reactions 1 assignee Claimed by @jviau View on GitHub
area: host-lifecycle bug needs-fix
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

## Description

When a language worker (e.g., Node.js) takes longer than `ProcessStartupTimeout` (default: 60 seconds) to send its `StartStream` gRPC message during the **first host startup**, the Functions Host enters a permanent deadlock and never recovers. The host remains stuck, returning 503 for every request until the process is manually restarted.

## Repro

Branch with full repro: [`shkr/repro-worker-startup-timeout-deadlock`](https://github.com/Azure/azure-functions-host/tree/shkr/repro-worker-startup-timeout-deadlock)

See [`test-apps/worker-startup-timeout-repro/README.md`](https://github.com/Azure/azure-functions-host/blob/shkr/repro-worker-startup-timeout-deadlock/test-apps/worker-startup-timeout-repro/README.md) for instructions. Quick start:

1. Check out the branch
2. Verify absolute paths in `src\WebJobs.Script.WebHost\Properties\launchSettings.json`
3. Set **WebJobs.Script.WebHost** as startup project → F5
4. `curl http://localhost:5050/api/hello` → **503 forever**

The repro uses a 5-second `processStartupTimeout` with an 8-second delay wrapper around the node worker to trigger the timeout.

## Potential root cause

Two issues combine to create a permanent deadlock:

### 1. `IsJobHostStarting()` returns `false` incorrectly

When `BuildHost()` throws during the first startup (e.g., worker timeout → `ExternalStartupException`), `ActiveHost` is never assigned (line 383 is never reached). This means `_currentJobHost` in `WorkerFunctionMetadataProvider` remains `null`.

On retry, `IsJobHostStarting()` checks `_currentJobHost is not null && State == Error` — this evaluates to `false` because `_currentJobHost` is `null`. The method incorrectly concludes the host is in a "final" error state rather than mid-startup retry.

This causes `GetFunctionMetadataAsync()` to call `RestartHostAsync()` (line 98) instead of `InitializeChannelAsync()` (line 91).

**File:** `WorkerFunctionMetadataProvider.cs`, lines 162-186

### 2. `RestartHostAsync()` deadlocks on `_hostStartSemaphore`

`RestartHostAsync()` is called from inside the retry's `BuildHost()` call chain. The call stack is:

```
StartHostAsync ← HOLDS _hostStartSemaphore (line 307)
→ UnsynchronizedStartHostCoreAsync
→ BuildHost()
→ FunctionMetadataManager
→ .GetFunctionMetadataAsync()
.GetAwaiter().GetResult() ← SYNC BLOCK (line 150)
→ WorkerFunctionMetadataProvider.GetFunctionMetadataAsync()
→ await RestartHostAsync()
→ await _hostStartSemaphore.WaitAsync() ← NEEDS the same semaphore
```

The `.GetAwaiter().GetResult()` at `FunctionMetadataManager.cs:150` blocks the thread synchronously. `RestartHostAsync` tries to acquire `_hostStartSemaphore` which is held by `StartHostAsync` on the same blocked thread. This creates a circular dependency that can never resolve.

**Files:**
- `FunctionMetadataManager.cs`, line 150 (`.GetAwaiter().GetResult()` sync-over-async)
- `WebJobsScriptHostService.cs`, line 627 (`_hostStartSemaphore.WaitAsync()` in `RestartHostAsync`)

## Timeline (from production telemetry)

| Time | Event |
|------|-------|
| 11:58:00.190 | Worker process startup initiated (60s timeout starts) |
| 11:59:00.190 | `PendingItem.OnTimeout()` fires (60s elapsed) |
| 11:59:00.264 | `StartStream` arrives (74ms too late, `MakeComplete()` returns false) |
| 11:59:00.615 | `TimeoutException` → `ExternalStartupException` |
| 11:59:08.336 | Host state → Error, `ActiveHost` never set (`'(null)' to '(null)'`) |
| 11:59:09.604 | Retry starts (op `f8119069`) |
| 11:59:09.664 | `IsJobHostStarting()` → **false** (bug) → calls `RestartHostAsync()` |
| 11:59:09.673 | Cancels retry op, then **deadlocks on semaphore** |
| 11:59:10+ | **Stuck permanently** — 503 on every request for 48+ hours |

## Impact

- Any worker startup timeout during first host startup permanently deadlocks the host. Host never recovers.

## Environment

- **Host version:** v4.1046.100 (also confirmed on `dev` branch as of 2/19/2026)
- **Runtime:** Node.js 14 (but not specific to node)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.