MSBuild server holds task/plugin assembly file handles for the process lifetime (no shadow-copy/unload)
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 141
Description
## Summary
The MSBuild server (`OutOfProcServerNode`, `nodemode:8`) is a persistent process that stays alive across builds. Today it releases *some* file handles between/after builds (the current directory and the runtime cache dir), but it **never unloads task/plugin/logger/SDK-resolver assemblies**, so any managed assembly loaded during a build stays loaded — and its file locked — for the entire lifetime of the server process. This is the classic "node reuse locks my task DLL" behavior, made more impactful by the server being longer-lived and shared across many builds/solutions.
This issue documents the current mechanism, the gap, how Roslyn's compiler server (`VBCSCompiler`) avoids the same problem, and options for improving it.
## What the server does today
Two deliberate handle-release steps exist:
1. **Current-directory handle.** On Windows a process holds a handle to its CWD, which would block deleting/renaming the just-built folder. After every build and at shutdown the server resets the CWD off the user's folder back to the tools dir:
- `src/Build/BackEnd/Node/OutOfProcServerNode.cs:469` (after each build)
- `src/Build/BackEnd/Node/OutOfProcServerNode.cs:262` (at shutdown)
- Worker nodes do the same: `src/Build/BackEnd/Node/OutOfProcNode.cs:495,722,727`
2. **Runtime cache dir.** Between reuse builds it deletes `%TEMP%\MSBuild-` via `FileUtilities.ClearCacheDirectory()` (`OutOfProcServerNode.cs:129`) to avoid stale/growing cache. It also disposes the redirected console writers and restores `Console.Out`/`Console.Error` each build (`OutOfProcServerNode.cs:449-465`).
Project files, imports, RAR reads, etc. are opened and closed, so those are not persistently held. The persistent-handle problem is specifically about **loaded managed assemblies**.
## The gap: loaded assemblies are never released
Task / logger / SDK-resolver / BuildCheck / project-cache assemblies loaded during a build stay loaded and locked for the whole life of the server:
- **On .NET:** `CoreClrAssemblyLoader` loads them via `MSBuildLoadContext`, which extends `AssemblyLoadContext` but is **non-collectible** — the constructor never passes `isCollectible: true` (`src/Framework/Loader/MSBuildLoadContext.cs:35-36`). Loaded assemblies are cached in `static` dictionaries that are never cleared (`src/Framework/Loader/CoreCLRAssemblyLoader.cs:25-26,113-124`; `src/Shared/TypeLoader.cs:30,56`).
- **On .NET Framework:** tasks without `[LoadInSeparateAppDomain]` load into the default AppDomain and stay (`src/Shared/TaskLoader.cs:128-134`); only separate-AppDomain tasks get unloaded (`TaskLoader.cs:168-178`).
Net effect: the server has the same locking behavior as a reused worker node, just longer-lived and shared across more builds. The most common pain is rebuilding a solution whose own build output is consumed as a task/analyzer/generator DLL — the server keeps the previous copy locked.
## How Roslyn's server (VBCSCompiler) avoids it
- **Shadow-copy** for analyzers/generators: `ShadowCopyAnalyzerAssemblyLoader` copies each analyzer DLL (+ dependencies/pdb) to a unique temp directory and loads from the copy, so the original build output is never locked and can be rebuilt while the server runs. On .NET Core it loads copies into a per-directory `DirectoryLoadContext`; on .NET Framework it's AppDomain + `LoadFrom` on the copy. Old shadow directories are cleaned up on startup.
- Reference assemblies / source files are read with delete-sharing and metadata is cached by timestamp, so no persistent exclusive locks are held.
## Options to improve
Ordered roughly by behavioral risk (lowest first):
1. **Shadow-copy task/plugin assemblies (Roslyn's model)** — proven, lowest behavioral risk. Scope it to non-SDK / build-output assemblies; keep SDK / `WellKnownAssemblyNames` in the default ALC (the loader already special-cases those in `MSBuildLoadContext.WellKnownAssemblyNames`). Cost: copy overhead + care around ALC identity and dependency resolution. Natural hook point: `CoreClrAssemblyLoader.LoadUsingPluginContext` / `MSBuildLoadContext`.
2. **Collectible ALC per build + unload** — releases handles cleanly, but tasks routinely leak static state / event handlers, and any GC-reachable object pins the context, so unload is unreliable; reloading common SDK tasks every build is also a perf hit.
3. **Do nothing / document** — rely on the out-of-proc task host for tasks that must not lock, and accept parity with existing node-reuse behavior.
**Suggested direction:** mirror Roslyn — opt-in shadow-copy limited to task/plugin assemblies loaded from outside the SDK/tools dirs, gated behind a ChangeWave / escape hatch, plus a server-reuse test that rebuilds a locked task DLL.
## Open questions
- Which concrete scenarios are hurting users most (rebuild-locking of output-as-task DLLs vs. others)?
- Should the fix also cover reused worker nodes (`OutOfProcNode`) and the task host, or is the server the priority?
- Interaction with existing escape hatches (`UseSingleLoadContext`, `UseCustomLoadContextForDependenciesInToolsDirectory`).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with CoreCLRAssemblyLoader.LoadUsingPluginContext and src/Framework/Loader/MSBuildLoadContext.cs, then review the existing cache and current-directory cleanup in OutOfProcServerNode.cs. Compare the issue's proposed shadow-copy approach with Roslyn's model and define a server-reuse test that rebuilds a task DLL while the server remains alive. Done should include an agreed scope and validation that the original task output is no longer locked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100