dotnet / dotnet/runtime

.NET 10.0.9 CoreCLR fatal error (0x80131506) in MSBuild multiproc node IPC on Linux

Open
#130,577 16 comments 1 reaction 1 assignee Claimed by @janvorli View on GitHub
area-VM-coreclr tracking-external-issue
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

## Summary

On .NET 10.0.9 (Linux x86_64), `dotnet build` (and `dotnet test`) of a multi-project solution intermittently crashes the host process with a CoreCLR fatal execution-engine error (`0x80131506` / "Internal CLR error"). A full minidump shows the crash is preceded by many `System.Net.Sockets.SocketException: "Cannot assign requested address /tmp/MSBuild"` thrown from MSBuild's out-of-proc node IPC (Unix domain sockets at `/tmp/MSBuild`), with faulting threads inside `Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.TryConnectToProcess`. Forcing a single in-proc MSBuild node (`-m:1`) eliminates the crash entirely (0/25 vs ~30% baseline).

## Environment

- **OS:** Linux x86_64 (Ubuntu, kernel 6.x). `/tmp` is `tmpfs`, `drwxrwxrwt`, 32 GB, plenty of free space/inodes.
- **.NET SDK:** `10.0.109`
- **.NET Runtime:** `Microsoft.NETCore.App 10.0.9` (`/usr/lib/dotnet/shared/Microsoft.NETCore.App`)
- **CoreCLR version (from `dotnet-dump` → `eeversion`):** `10.0.926.27113` @ commit `901ca941248413c79832d2fdbd709da0c4386353`
- **SOS version:** `9.0.13.11903`
- GC mode: Workstation (default)
- Only one SDK installed (no `global.json` pinning to an older version)

## Reproduction

A solution of ~11 C# projects (mix of class libraries and test projects, .NET 10.0). In a loop:

```
for i in $(seq 1 30); do dotnet build MySolution.slnx || echo "crashed on iter $i"; done
```

Within ~20 iterations the process aborts:

```
Fatal error.
Internal CLR error. (0x80131506)
```

### Observed crash rates (this solution, Linux)

| Command | Crashes | Total |
|---|---|---|
| `dotnet build Virelia.slnx` (11 projects) | 6 | 20 |
| `dotnet test Virelia.slnx` (same solution) | 2 | 15 |
| `dotnet build`/`test` of a **3-project** solution | 0 | 32 |
| `dotnet build Virelia.slnx -m:1` (single in-proc node) | 0 | 25 |
| `dotnet build Virelia.slnx -nodereuse:false` (still multiproc) | 5 | 15 |

So the crash correlates with **multiproc out-of-proc node spawning** (larger solutions spawn more nodes): a 3-project solution did not reproduce in 32 iterations, while the 11-project solution crashes ~30% of the time. `-m:1`, which disables out-of-proc nodes entirely, eliminates it (0/25).

### Downstream symptom (possibly a separate test-runner bug)

When the crash hits during `dotnet test`, the faulted process sometimes **wedges** (the runtime's `createdump` reports `Target process is alive` after writing the dump, and the process does not exit). A foreground `dotnet test` invocation then appears to **hang** — a spawned child (testhost) inherits the stdout pipe and keeps it open, so the caller never sees EOF. Under a clean harness (process in its own session, stdout to a file), `dotnet test` exits on every crash (2 crash-exit, 0 hang in 15). I have not fully isolated whether the hang is an independent `dotnet test`/vstest robustness bug or purely a side effect of the wedged process + inherited pipe; flagging it here in case the test team wants to harden the runner against a dead/wedged build child.

## Analysis (from the full minidump)

Captured via the runtime's built-in dumper (`DOTNET_DbgEnableMiniDump=1`, `DOTNET_DbgMiniDumpType=4`), analyzed with `dotnet-dump analyze`:

1. **No current managed exception** on the main thread — the fault is a native CoreCLR execution-engine error. `0x80131506` corresponds to `System.ExecutionEngineException`.
2. `dumpexceptions` shows the pre-allocated EE/stackoverflow/OOM exceptions on the heap **plus many** `System.Net.Sockets.SocketException: Cannot assign requested address /tmp/MSBuild`, several carrying the stack frame `System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)`.
3. Thread 0 (the `dotnet` → MSBuild forwarder) is in `Microsoft.Build.Execution.BuildSubmission.Execute()` → `WaitOneNoCheck`.
4. Several worker threads are inside `Microsoft.Build.BackEnd.NodeProviderOutOfProcBase`:
- `TryConnectToProcess` → `TryConnectToPipeStream` → `NamedPipeClientStream.ConnectInternal` + `SpinWait`. (Despite the `NamedPipeClientStream` type name, on Unix this is Unix-domain-socket IPC at `/tmp/MSBuild`.)
- reached via `System.Threading.Tasks.Parallel.For` over `GetNodes`/`StartNewNode` — parallel out-of-proc node spawning.
5. One thread is in `Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests()`.
6. Each crash leaves ~10 stale `/tmp/MSBuild` socket files behind (the node IPC endpoints), confirming the sockets were created but never cleaned up because the process faulted. (Cleaning these between runs does **not** prevent the next crash — the litter is a symptom, not the cause.)

## Expected behavior

The .NET runtime must not crash with a fatal execution-engine error (`0x80131506`) regardless of what managed code does with a socket. A `SocketException` (`EADDRNOTAVAIL` on a Unix domain socket) is a normal, recoverable managed failure and must be surfaced as a managed exception, never race the runtime into an unrecoverable CoreCLR fault.

## Workaround

`-m:1` (force a single in-proc MSBuild node) eliminates the crash (0/25 in my testing) by never spawning out-of-proc nodes and thus never creating/connecting the `/tmp/MSBuild` sockets. No perceptible perf cost for this solution (build/test times stay ~1–3s).

## Related

- **dotnet/msbuild#13718** — "Fail fast on connect-to-node when launched child process has already exited" — an open design proposal for exactly the `NodeProviderOutOfProcBase.TryConnectToPipeStream` code path this crash hits. On .NET 10.0.9 the failure manifests not just as the 30s hang that issue targets, but as a CoreCLR fatal crash.
- **dotnet/sdk#52579** — testhost crash regression after 10.0.100→10.0.102 (different trigger: >16-param source-generated delegates), same "passes then aborts" symptom.

## Minidump

A **full** CoreCLR minidump (467 MB raw → 66 MB zstd-compressed), produced by the runtime's `createdump` on the `0x80131506` fault, is available for download:

- **Download:** https://github.com/estrizhok/repro/releases/download/dotnet-runtime-0x80131506-dump/prog_full.zst
- **Decompress:** `zstd -d prog_full.zst` (yields `prog_dotnet_pid_558566.dmp`, ~446 MB)
- **Analyze:** `dotnet-dump analyze prog_dotnet_pid_558566.dmp`

`dotnet-dump analyze` loads it cleanly; `eeversion` confirms CoreCLR `10.0.926.27113 @ commit 901ca941248413c79832d2fdbd709da0c4386353`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.