Azure / Azure/azure-functions-dotnet-worker

FunctionsNetHost exits with code 139 (SIGSEGV) on .NET 10 isolated worker at normal operating load on Flex Consumption (Linux)

Open
#3,335 16 comments 5 reactions 2 assignees Claimed by @kshyju View on GitHub
.NET 10
Dominant language
C#
Stars
466
Forks
215
Avg merge
3d 10h
Merged PRs (30d)
7

Description

### Description

On **.NET 10 isolated worker** running on **Flex Consumption (Linux)**, `FunctionsNetHost` crashes with exit code **139 (0x8B)** — a SIGSEGV (segmentation fault). The crashes occur at **normal, routine production load** — not under stress conditions — and repeat in rapid succession, destabilizing the function app for 6–10 minute windows.

The configured concurrency at time of crash:
- **`maxConcurrentSessions: 3`** — the Service Bus session processor ceiling, not a peak
- **`maxConcurrentCalls: 16`**
- **`maxMessageBatchSize: 1`**

No application code exception precedes the crash. The worker process simply terminates with SIGSEGV. The Azure Functions HOST process (`HostInstanceId` is stable throughout) detects the crash and restarts the worker, but under continued normal load the worker crashes again within 1–2 minutes.

The crashes affect **two separate function apps** in the same environment on the same day.

---

## Exception / Error Log

### GroupUpdaterFunction (GraphUpdater function app)

```
Exception while executing function: Functions.GroupUpdaterFunction
Microsoft.Azure.WebJobs.Host.FunctionInvocationException
---> Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException:
/azure-functions-host/workers/dotnet-isolated/bin/FunctionsNetHost exited with code 139 (0x8B)
---> System.Exception: n/a

at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.ThrowIfExitError
(WorkerProcess.cs:233) [Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1047.0.0]
at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.OnProcessExited
(WorkerProcess.cs:185) [Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1047.0.0]
at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+d__9.MoveNext
(WorkerFunctionInvoker.cs:103) [Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1047.0.0]
```

### DeltaLinkUploaderFunction → SubOrchestratorFunction → OrchestratorFunction (GroupMembershipObtainer function app)

```
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
Exception while executing function: Functions.SubOrchestratorFunction
---> DurableTask.Core.Exceptions.OrchestrationFailureException:
Task 'DeltaLinkUploaderFunction' (#8) failed with an unhandled exception:
Exception while executing function: Functions.DeltaLinkUploaderFunction
[Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=3.0.0.0]
```

---

## Crash Events

### Event 1 — GraphUpdater function app

**Date/time window:** 2026-03-05, 16:53 – 17:02 UTC
**Worker restarts:** 16:53:01 · 16:55:41 · 16:57:01 · 16:59:51 · 17:00:41 · 17:01:41 UTC
**Duration of instability:** ~9 minutes, 6 worker crashes

| Identifier | Value |
|---|---|
| `HostInstanceId` | `b83e8852-05dd-4c8b-9631-4b23867e7342` (stable throughout — HOST did not crash) |
| DTF Orchestration RunId | `bfac744d-0c12-5729-8d64-7f6be374114b` |

### Event 2 — GroupMembershipObtainer function app

**Date/time:** 2026-03-05, 23:20 – 23:21 UTC
Two concurrent compute instances affected simultaneously.

**Instance A** — DeltaLinkUploaderFunction → SubOrchestratorFunction crashed

| Identifier | Value |
|---|---|
| Timestamp | `2026-03-05T23:20:54.4169845Z` |
| `HostInstanceId` | `2bdf0b18-35f9-4e87-a511-bbeeb759df7a` |
| `cloud_RoleInstance` | `0--de8f0470-1b46-41c7-89de-3c813d91e906` |
| `ProcessId` | `34` |
| `InvocationId` (SubOrchestratorFunction) | `4d3ae322-a843-4003-9c16-41e5590c1ead` |
| `operation_Id` | `7ce913159e53623eb759b626fd6314f6` |

**Instance B** — OrchestratorFunction failure propagated from same root crash

| Identifier | Value |
|---|---|
| Timestamp | `2026-03-05T23:20:59.7993100Z` |
| `HostInstanceId` | `efe7cf84-3f6f-4418-acc2-07d1cb9096c0` |
| `cloud_RoleInstance` | `0--c133ae49-0590-43a4-ac05-a30950dd3ce4` |
| `ProcessId` | `34` |
| `InvocationId` (OrchestratorFunction) | `1c1ae40f-af9a-499d-adc3-0853781db8e1` |
| `operation_Id` | `2afc24c3598368fc87747be9de4b323f` |

### Event 3 — GraphUpdater function app (same day, later)

**Date/time:** 2026-03-05T23:26:49.9042201Z

| Identifier | Value |
|---|---|
| Timestamp | `2026-03-05T23:26:49.9042201Z` |
| `HostInstanceId` | `b99961179-d28e-48b3-aabf-65ac5d9e1f54` |
| `ProcessId` | `35` |
| `InvocationId` (GroupUpdaterFunction) | `0520f145-bc19-4004-bc8d-a0fa8611b768` |
| Invocation duration at time of crash | `46,155 ms` |

---

## Steps to Reproduce

Not deterministically reproducible on demand, but the confirmed runtime conditions at time of all crashes:

1. Flex Consumption plan, Linux, .NET 10 isolated worker
2. Durable Functions active (`extendedSessionsEnabled: true`, 30s idle timeout, `maxConcurrentSessions: 3`)
3. Service Bus session-based trigger processing at its configured ceiling (`maxConcurrentSessions: 3`)
4. Under these routine operating conditions, `FunctionsNetHost` crashes with SIGSEGV — no exception in application code precedes it

---

## Expected Behavior

The worker process should not crash with a segmentation fault under routine production load. If a crash does occur, it should not recur repeatedly within the same operating window.

---

## Actual Behavior

`FunctionsNetHost` crashes with exit code 139 (0x8B / SIGSEGV) repeatedly. The Azure Functions host detects the crash, restarts the worker, but under continued normal load the worker crashes again within 1–2 minutes. The cycle repeats 6 times over ~9 minutes before stabilizing.

---

## Downstream Impact: Durable Entity State Loss

Because `FunctionsNetHost` terminates mid-execution, in-flight Durable Entity signals are orphaned — the entity dispatch is sent to the worker but the worker crashes before completing, leaving no `ContinuedAsNew` response written to Table Storage.

Observed sequence (Event 1, all times UTC):

| Time | Sequence | Event |
|---|---|---|
| `2026-03-05T16:56:02Z` | seq 1726 | Entity signal dispatched → **no completion log** (orphaned dispatch) |
| `2026-03-05T16:56:05Z` | seq 1731 | Entity state committed via `ContinuedAsNew:True` with `{MessagesProcessed:3, IsValidGroup:true}` |
| `2026-03-05T16:57:06Z` | seq 1851 | Entity re-executed in **~1ms** — `InitializeState()` called (fresh/null state), not loaded from Table Storage — 61 seconds after confirmed write |

The state loss cascades: only 2 of 5 expected `MessagesProcessed` increments are counted → business-level sync job failure ("Not all messages were processed, only 2 out of 5").

---

## Environment

| Property | Value |
|---|---|
| OS | Linux |
| Azure Region | `westus2` |
| Hosting plan | Flex Consumption |
| Worker model | Isolated (out-of-process) |
| Worker target framework | `net10.0` |
| .NET SDK (global.json) | `10.0.103` (`rollForward: latestFeature`) |
| .NET Runtime | `10.0.3` |
| Azure Functions host SDK | `azurefunctions: 4.1047.100.26071` |
| `Microsoft.Azure.WebJobs.Script.Grpc` | `4.1047.0.0` |
| `Microsoft.Azure.WebJobs.Script` | `4.1047.0.0` |
| `Microsoft.Azure.WebJobs.Host` | `3.0.44.0` |
| `Microsoft.Azure.WebJobs.Extensions.DurableTask` (host-side) | `3.0.0.0` |
| `Microsoft.Azure.Functions.Worker` (NuGet) | `2.51.0` |
| `Microsoft.Azure.Functions.Worker.Sdk` (NuGet) | `2.0.7` |
| `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` (NuGet) | `1.16.0` |
| `Microsoft.DurableTask.Abstractions` (NuGet) | `1.22.0` |
| `maxConcurrentSessions` | `3` (configured ceiling, not a stress peak) |
| `maxConcurrentCalls` | `16` |
| `extendedSessionsEnabled` | `true` (30s idle timeout) |
| `functionTimeout` | `00:20:00` |

---

## Additional Context

[host.json](https://github.com/user-attachments/files/25805576/host.json)

[crash-instance-1-20260305-232054-unknown-op-sanitized.json](https://github.com/user-attachments/files/25805614/crash-instance-1-20260305-232054-unknown-op-sanitized.json)
[crash-instance-2-20260305-232054-unknown-op-sanitized.json](https://github.com/user-attachments/files/25805613/crash-instance-2-20260305-232054-unknown-op-sanitized.json)
[crash-instance-2-20260305-232059-unknown-op-sanitized.json](https://github.com/user-attachments/files/25805616/crash-instance-2-20260305-232059-unknown-op-sanitized.json)
[crash-instance-3-20260305-232059-unknown-op-sanitized.json](https://github.com/user-attachments/files/25805615/crash-instance-3-20260305-232059-unknown-op-sanitized.json)

- The HOST process remains stable throughout all crashes (constant `HostInstanceId` per event) — only `FunctionsNetHost` (the dotnet-isolated worker) crashes
- The crash is a raw SIGSEGV with no managed exception preceding it — no application-level exception handler fires before the process terminates
- The crashes affect **two independent function apps** in the same environment
- **Three separate crash events** occurred on 2026-03-05, spanning two function apps across two time windows (16:53–17:02 UTC and 23:20–23:27 UTC)
- Also observed on 2026-03-06
- Subscription ID and function app resource ID can be provided privately on request — not included here to avoid exposing sensitive identifiers publicly

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.