dotnet / dotnet/msbuild

dotnet build-server shutdown does not shut down sidecar taskhosts

Closed
#14,313 2 comments 0 reactions 1 assignee Claimed by @JanProvaznik View on GitHub
triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Summary

In multithreaded mode (`-mt`), non-enlightened tasks run in **reusable sidecar TaskHost** processes. `dotnet build-server shutdown` does **not** terminate them (neither does `BuildManager.ShutdownAllNodes()`). They keep running until the OS session ends. The MSBuild Server, out-of-proc worker nodes, and Framework `MSBuildTaskHost.exe` hosts **are** reaped correctly — only the `dotnet`-hosted MT sidecars leak.

### Repro

1. `dotnet build MySolution.sln -t:restore -mt` (restore pulls in non-enlightened NuGet tasks).
2. `dotnet build-server shutdown`.
3. **Expected:** the sidecar TaskHost processes exit. **Actual:** they keep running; the next `-mt` build reconnects to the same PIDs.

### Evidence

Across two consecutive `-mt` restore builds separated by `dotnet build-server shutdown`, the **owner (caller) process changed** (8376 → 3592) while **56/57 sidecar TaskHost PIDs stayed identical**. Sidecars are independent processes that survive their owner's exit (node-reuse design) and have no idle timeout ([taskhost-threading spec](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/documentation/specs/multithreading/taskhost-threading.md)); nothing ever signals them to stop.

### Root cause

`dotnet build-server shutdown` → `MSBuildServer.Shutdown()` (sdk) → [`BuildManager.ShutdownAllNodes()`](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/BuildManager/BuildManager.cs#L1455-L1461) → [`NodeManager.ShutdownAllNodes()`](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/Components/Communications/NodeManager.cs#L149-L153), which only shuts down the **worker** provider. The `TaskHostNodeManager` / `NodeProviderOutOfProcTaskHost` is never invoked. Two further gaps mean even the worker path couldn't reach them:

- [`NodeProviderOutOfProcBase.ShutdownAllNodes`](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs#L161-L198) only name-scans Framework `MSBuildTaskHost.exe` ([L172](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs#L172)) and connects with the **worker** handshake ([L181](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs#L181)) — never the sidecar handshake (`HandshakeOptions.SidecarTaskHost`, [HandshakeOptions.cs#L59](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Framework/BackEnd/HandshakeOptions.cs#L59)).
- The in-build cascade ([`OutOfProcNode.HandleShutdown` → `_taskHostNodeManager.ShutdownConnectedNodes`](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/Build/BackEnd/Node/OutOfProcNode.cs#L492)) can't help: the sidecars' owner has already exited, so no live node holds a connection to cascade through.

### Proposed fix

1. **Dispatch:** route `BuildManager.ShutdownAllNodes()` through `TaskHostNodeManager` / `NodeProviderOutOfProcTaskHost` as well.
2. **Discovery:** give that provider a real `ShutdownAllNodes` that enumerates `dotnet`-hosted sidecars and connects with the sidecar handshake (`TaskHost | NET | SidecarTaskHost | | NodeReuse`), rather than the worker handshake / Framework-only scan.
3. **Honor the signal:** a sidecar must actually exit on an explicit teardown. Today [`OutOfProcTaskHostNode.HandleNodeBuildComplete`](https://github.com/dotnet/msbuild/blob/3d4b39104e9623bded78eb36bc9677dc422ebc0b/src/MSBuild/OutOfProcTaskHostNode.cs#L1331-L1340) always chooses `BuildCompleteReuse` when `_nodeReuse` is set, ignoring `buildComplete.PrepareForReuse == false` — so "shut down like an in-build node shutdown" alone won't work. It must honor a non-reuse request (or use a dedicated terminate signal):
```csharp
if (_nodeReuse && buildComplete.PrepareForReuse)
_shutdownReason = NodeEngineShutdownReason.BuildCompleteReuse;
else
_shutdownReason = (buildComplete.PrepareForReuse && Traits.Instance.EscapeHatches.ReuseTaskHostNodes)
? NodeEngineShutdownReason.BuildCompleteReuse
: NodeEngineShutdownReason.BuildComplete;
```

### Environment

MSBuild `18.10.0-preview-26358-03` (observed); root cause on `main` @ `3d4b39104e9623bded78eb36bc9677dc422ebc0b`. Windows, `-mt`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.