BuildHost crashes with unhandled `IOException: Pipe is broken` when writing an RPC response after the caller has already disconnected
- Dominant language
- C#
- Stars
- 20.7k
- Forks
- 4.3k
- PR merge metrics
- PR metrics pending
Description
**Version Used**:
`roslyn-language-server` `5.8.0-1.26266.2` and `5.10.0-1.26355.102` (both exhibit the same failure; reproduced on the latter after an in-place version upgrade specifically to rule out an already-fixed bug).
Windows 11, .NET SDK matching the pinned Roslyn `global.json` version during local fix verification (`10.0.109`).
**Steps to Reproduce**:
1. Point a custom LSP client (not Visual Studio or VS Code - a standalone client driving `roslyn-language-server` directly over stdio) at a real, complex, multi-project C# solution that uses `MSBuildWorkspace`/`BuildHost.exe` for project loading.
2. Drive a sustained sequence of `textDocument/references`, `textDocument/implementation`, `textDocument/prepareCallHierarchy`, and `callHierarchy/outgoingCalls` requests across the solution's symbols (walking every symbol in the workspace, not a single hot path).
3. Continue for roughly 40-450 requests (the exact threshold varies run to run). The crash always occurs mid-`textDocument/references`, but retrying the exact same request in isolation afterward succeeds - it is not tied to a specific symbol or file, which points to some form of accumulated state/resource exhaustion inside `BuildHost.exe` rather than a specific input triggering it.
A minimal, fully isolated repro (e.g. a small solution + a script that reliably reproduces this in under a minute) is not available yet - the crash requires sustained real-world request volume against a real solution to trigger whatever causes the RPC caller to give up on the connection first. See "Investigation" below for what was ruled out instead.
**Diagnostic Id**:
Not applicable - this is not an analyzer diagnostic, it's an unhandled exception that terminates the `BuildHost.exe` process.
**Expected Behavior**:
A downstream RPC server should never let an already-disconnected client crash the whole host process. If `BuildHost.exe` finishes processing a request and the caller has since given up and closed its end of the pipe, the resulting write failure should be treated as an expected, ignorable condition (there's no one left to receive the response), logged, and the process should continue running normally.
**Actual Behavior**:
`BuildHost.exe` terminates immediately with an unhandled exception. Depending on how the calling process (`Microsoft.CodeAnalysis.LanguageServer`, the VS Code C# extension, or any other consumer of `roslyn-language-server`) handles a `BuildHost` process dying mid-operation, this can destabilize or crash the whole language server session rather than failing just the one in-flight request.
## Crash dump analysis
Captured via Windows Error Reporting (`%LOCALAPPDATA%\CrashDumps`) and confirmed with `dotnet-dump analyze -c "pe -lines" -c "exit"`:
```
Exception type: System.IO.IOException
Message: Pipe is broken.
StackTrace (generated):
System.Private.CoreLib!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Private.CoreLib!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Private.CoreLib!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(...)
System.Private.CoreLib!System.IO.StreamWriter+<g__Core|79_0>d.MoveNext()
Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost!Microsoft.CodeAnalysis.MSBuild.RpcServer+d__11.MoveNext()
Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost!Microsoft.CodeAnalysis.MSBuild.RpcServer+d__10.MoveNext()
Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost!Microsoft.CodeAnalysis.MSBuild.Program+d__0.MoveNext()
```
A separate, targeted diagnostic build (temporarily instrumented to capture the exact exception shape at the point of the throw, rather than relying on the WER dump alone) confirmed:
```
Type=System.IO.IOException HResult=0x800700E8 Message=Pipe is broken.
```
with no `InnerException` at all (depth 0). `0x800700E8` is `ERROR_NO_DATA` ("The pipe is being closed"), consistent with the RPC caller having already closed its end of a named pipe before `BuildHost.exe`'s response write completes.
## Investigation: ruling out alternative causes
Because this was first observed as an intermittent crash of an entire downstream Node.js-based tool (not a Roslyn crash) that happened to depend on `roslyn-language-server`, considerable effort went into ruling out causes other than Roslyn itself before concluding this is a genuine upstream bug:
- **Not the downstream client's process/IPC architecture.** The downstream tool's language-server process management was substantially reworked (moved to an isolated worker process, then to a different IPC transport entirely - a TCP loopback socket instead of a named pipe/`fork()` channel) specifically to try to contain or avoid this failure. Neither change had any effect: the crash reproduced identically regardless of the downstream process topology or IPC transport used to talk to `roslyn-language-server`.
- **Not request pacing/timing.** Adding a deliberate delay (250ms) before each symbol's batch of requests - roughly 4-5x slower request issuance than the unpaced run - did not prevent the crash, which still occurred at a similar request count. This rules out "the caller just needs to slow down" as a workaround and points toward the failure being driven by cumulative request/resource count, not throughput.
- **Not Windows Defender.** Real evidence surfaced during this investigation that Defender's behavioral engine had flagged an unrelated helper process as "Suspicious Behavior" during testing, which looked like a plausible confound. Tested directly: adding Defender process-scan exclusions for the relevant executables changed the crash threshold somewhat (inconclusive on its own), but fully disabling Defender Behavior Monitoring machine-wide reproduced the crash again at almost the identical request threshold as with just the exclusions in place. Two different Defender configurations landing at the same threshold rules out Defender as the cause.
- **Not a testing-environment artifact.** The crash was independently reproduced by a second person, on a separate machine, driving the same workflow entirely outside the original testing tool/environment (a plain PowerShell session, not through any wrapping shell or automation harness) - identical failure signature: silent process death partway through a build, exit code 1, no further output.
- **Decisive control experiment isolating the fault to Roslyn specifically.** The same benchmark harness and process/IPC architecture was run against a synthetic TypeScript workspace (300 generated files, cross-referencing functions/classes) using `typescript-language-server` instead of `roslyn-language-server`, issuing an equivalent-or-greater request volume. Result: clean exit code 0, proper shutdown, roughly 4-5x more total requests processed than Roslyn ever survived before crashing. Since the only variable changed between the two runs was which language server binary was spawned, this isolates the fault specifically to Roslyn/`dotnet.exe`/`BuildHost.exe`, not the downstream client's architecture, Windows, or Node's process/pipe handling in general.
- **Direct proof the `BuildHost`-adjacent process is terminated by an unhandled exception, not gracefully.** Diagnostic logging added to the downstream client's worker process (covering every request's start/success/failure and all of `uncaughtException`/`unhandledRejection`/`exit`) showed the log stream simply stopping mid-request with zero further output of any kind, including no exit-handler firing on the client side - consistent with the `BuildHost.exe` side dying from the unhandled exception shown in the crash dump above, taking down the pipe (and therefore the client's view of the connection) abruptly rather than as part of any graceful shutdown sequence.
## Root cause
`src/Workspaces/MSBuild/BuildHost/Rpc/RpcServer.cs`, `ProcessRequestAsync`, response-write block:
```csharp
using (await _sendingStreamSemaphore.DisposableWaitAsync().ConfigureAwait(false))
{
await _streamWriter.WriteLineAsync(responseJson).ConfigureAwait(false);
await _streamWriter.FlushAsync().ConfigureAwait(false);
}
```
There is no try/catch around the write. If the underlying pipe is already broken by the time this runs, `FlushAsync()` throws `IOException`. Nothing catches it: the surrounding call is inside a fire-and-forget `Task.Run(() => ProcessRequestAsync(request))` in `RunAsync()`, which later does `await Task.WhenAll(remainingTasks)` on all such tasks with no surrounding try/catch, so the exception propagates uncaught all the way to `Program.Main`, crashing the entire `BuildHost.exe` process.
This is a different bug from #77040 / PR #77151, which fixed a broken-pipe exception on the *client* side (`BuildHostProcessManager.BuildHostProcess.DisposeAsync()`), already caught and logged there as a `WorkspaceFailed` event. This issue is about the *server* side, inside `BuildHost.exe` itself, crashing unhandled - a different code path entirely.
## Proposed fix
Wrap the response write in a try/catch that logs and swallows `IOException`, following the same broad-catch-and-log convention already used elsewhere in this codebase for "the other side of a pipe/connection is gone" scenarios (e.g. `BuildHostProcessManager.DisposeAsync()`, `BuildServerConnection.cs`), rather than trying to enumerate every specific HResult that can mean "client disconnected." A `BuildHostLogger?` is threaded through `RpcServer`'s constructors (optional, defaulting to `null`, to avoid breaking any other callers) so the failure is still visible in logs rather than silently disappearing.
I have this fix implemented and locally verified (including a new unit test that deterministically forces the client-already-disconnected ordering, rather than relying on the race that manifests in real-world use) on a fork, and I'm planning to open a PR referencing this issue.
Contributor guide
Research direction
Start in src/Workspaces/MSBuild/BuildHost/Rpc/RpcServer.cs, especially RunAsync and ProcessRequestAsync, then compare the pipe-error handling in BuildHostProcessManager.DisposeAsync and BuildServerConnection.cs. Run the new deterministic unit test mentioned in the issue. Done means a disconnected client is logged without an unhandled exception, and BuildHost.exe remains running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100