Azure / Azure/azure-functions-host
Host deadlock after worker startup timeout
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
## Description
When a language worker (e.g., Node.js) takes longer than `ProcessStartupTimeout` (default: 60 seconds) to send its `StartStream` gRPC message during the **first host startup**, the Functions Host enters a permanent deadlock and never recovers. The host remains stuck, returning 503 for every request until the process is manually restarted.
## Repro
Branch with full repro: [`shkr/repro-worker-startup-timeout-deadlock`](https://github.com/Azure/azure-functions-host/tree/shkr/repro-worker-startup-timeout-deadlock)
See [`test-apps/worker-startup-timeout-repro/README.md`](https://github.com/Azure/azure-functions-host/blob/shkr/repro-worker-startup-timeout-deadlock/test-apps/worker-startup-timeout-repro/README.md) for instructions. Quick start:
1. Check out the branch
2. Verify absolute paths in `src\WebJobs.Script.WebHost\Properties\launchSettings.json`
3. Set **WebJobs.Script.WebHost** as startup project → F5
4. `curl http://localhost:5050/api/hello` → **503 forever**
The repro uses a 5-second `processStartupTimeout` with an 8-second delay wrapper around the node worker to trigger the timeout.
## Potential root cause
Two issues combine to create a permanent deadlock:
### 1. `IsJobHostStarting()` returns `false` incorrectly
When `BuildHost()` throws during the first startup (e.g., worker timeout → `ExternalStartupException`), `ActiveHost` is never assigned (line 383 is never reached). This means `_currentJobHost` in `WorkerFunctionMetadataProvider` remains `null`.
On retry, `IsJobHostStarting()` checks `_currentJobHost is not null && State == Error` — this evaluates to `false` because `_currentJobHost` is `null`. The method incorrectly concludes the host is in a "final" error state rather than mid-startup retry.
This causes `GetFunctionMetadataAsync()` to call `RestartHostAsync()` (line 98) instead of `InitializeChannelAsync()` (line 91).
**File:** `WorkerFunctionMetadataProvider.cs`, lines 162-186
### 2. `RestartHostAsync()` deadlocks on `_hostStartSemaphore`
`RestartHostAsync()` is called from inside the retry's `BuildHost()` call chain. The call stack is:
```
StartHostAsync ← HOLDS _hostStartSemaphore (line 307)
→ UnsynchronizedStartHostCoreAsync
→ BuildHost()
→ FunctionMetadataManager
→ .GetFunctionMetadataAsync()
.GetAwaiter().GetResult() ← SYNC BLOCK (line 150)
→ WorkerFunctionMetadataProvider.GetFunctionMetadataAsync()
→ await RestartHostAsync()
→ await _hostStartSemaphore.WaitAsync() ← NEEDS the same semaphore
```
The `.GetAwaiter().GetResult()` at `FunctionMetadataManager.cs:150` blocks the thread synchronously. `RestartHostAsync` tries to acquire `_hostStartSemaphore` which is held by `StartHostAsync` on the same blocked thread. This creates a circular dependency that can never resolve.
**Files:**
- `FunctionMetadataManager.cs`, line 150 (`.GetAwaiter().GetResult()` sync-over-async)
- `WebJobsScriptHostService.cs`, line 627 (`_hostStartSemaphore.WaitAsync()` in `RestartHostAsync`)
## Timeline (from production telemetry)
| Time | Event |
|------|-------|
| 11:58:00.190 | Worker process startup initiated (60s timeout starts) |
| 11:59:00.190 | `PendingItem.OnTimeout()` fires (60s elapsed) |
| 11:59:00.264 | `StartStream` arrives (74ms too late, `MakeComplete()` returns false) |
| 11:59:00.615 | `TimeoutException` → `ExternalStartupException` |
| 11:59:08.336 | Host state → Error, `ActiveHost` never set (`'(null)' to '(null)'`) |
| 11:59:09.604 | Retry starts (op `f8119069`) |
| 11:59:09.664 | `IsJobHostStarting()` → **false** (bug) → calls `RestartHostAsync()` |
| 11:59:09.673 | Cancels retry op, then **deadlocks on semaphore** |
| 11:59:10+ | **Stuck permanently** — 503 on every request for 48+ hours |
## Impact
- Any worker startup timeout during first host startup permanently deadlocks the host. Host never recovers.
## Environment
- **Host version:** v4.1046.100 (also confirmed on `dev` branch as of 2/19/2026)
- **Runtime:** Node.js 14 (but not specific to node)
Contributor guide
Assessment
This issue has not been assessed yet.