dotnet / dotnet/roslyn

macOS: `DefaultFileChangeWatcher`'s consolidate-on-cap design still pays for many serialised native watcher starts before it fires

Open
#84,953 4 comments 0 reactions 1 assignee Claimed by @jasonmalinowski View on GitHub
Area-IDE
Dominant language
C#
Stars
20.7k
Forks
4.3k
PR merge metrics
PR metrics pending

Description

### Summary

On macOS, `DefaultFileChangeWatcher` consolidates a directory's watchers only once the process-wide watcher count reaches its cap (50 on non-Windows, set by `DelegatingFileChangeWatcher`). Below that cap, every reference directory gets its own precise, native FSEvents watcher, and each start is expensive and fully serialised on a single thread. On a solution with fewer than 100 reference directories, I measured this registration work alone at 84 to 86 percent of a 4.9 second solution load phase.

I believe this is a direct, macOS-specific follow-up to dotnet/roslyn#83101 rather than a duplicate of it. That PR fixed watcher exhaustion (the Linux inotify problem from dotnet/roslyn#82857) by capping and consolidating. On macOS the cost is not exhaustion: it is the per-watcher start latency paid on the way to the cap.

### Environment

- macOS 27.0 (Darwin 27.0.0), Apple M1 Pro, arm64
- .NET 10.0.8
- `Microsoft.CodeAnalysis.LanguageServer`, run headless over stdio, not inside Visual Studio or VS Code

### What I measured

Two runs of a thread-time trace over the solution load phase, both isolated (idle machine, no other builds running):

- Run 1: 83.7 percent of the phase inside watcher registration. Run 2: 86.1 percent.
- The remainder is small and accounted for: metadata reads about 0.3 percent, parsing about 3.5 ms, MSBuild evaluation near zero.
- The fixture referenced 96 distinct directories (plus the source tree). Registering all 96 took about 6856 ms started in sequence.
- I tried starting the watchers from multiple threads as an experiment. It did not help: about 7208 ms for the same 96, slightly worse than sequential. That pointed at a shared serialising resource, and the native captures later named it (next two points).
- The per-start cost measures 53 to 75 ms per `EnableRaisingEvents` across my runs (65 ms mean by direct probe). That is consistent in order of magnitude with dotnet/runtime#77793 (about 45 to 50 ms per start on Apple silicon), fixed at the runtime level by dotnet/runtime#121698. That runtime fix had not reached the .NET 10.0.8 SDK I tested against, and it lowers the constant rather than removing the linear ramp.
- I corroborated the managed trace with native `sample` captures, and they corrected my first guess about the mechanism. The hot frames are not `FSEventStreamStart`'s own Mach RPC to `fseventsd`: they sit under `sync`, a whole-filesystem flush reached on every watcher start. On this machine a bare `sync()` measures 64.8 ms median and `EnableRaisingEvents` measures 64.3 ms: the same cost. This also explains the failed multi-thread experiment above, because `sync` is a global barrier and concurrent starts serialise by construction. It suggests the per-start constant is itself attackable at the runtime level, independently of the consolidation design this issue is about.
- A second, smaller cost I found while tracing: `ExpandWatcherToCover` flipping `IncludeSubdirectories` from false to true measures at about 78 ms, because that specific change forces a full FSEventStream teardown and rebuild. Adding a filter to an already-recursive watcher does not restart the stream (.NET matches filters in managed code per event on macOS); only the recursion flip does. This is a second reason precise-then-widen costs more than starting recursive where a directory is already known to need it.
- The same registration cost is very likely paid on cold loads too; on my fixture a cold load runs about 5.7 seconds.

### Mechanism

1. `ReferenceFileChangeTracker` (`src/Workspaces/Core/Portable/Workspace/ProjectSystem/ReferenceFileChangeTracker.cs`) declares one `WatchedDirectory` per distinct reference directory.
2. `DefaultFileChangeWatcher` (`src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/FileWatching/DefaultFileChangeWatcher.cs`) starts one native watcher per directory, precise (non-recursive) by default.
3. `FindBestNodeToConsolidateAndConsolidateIt_NoLock` runs its consolidation pass only once `_currentWatcherCount >= _maxWatcherCount`, and `DelegatingFileChangeWatcher` sets that cap to 50 on non-Windows.
4. A solution with, say, 60 reference directories under one parent therefore pays for roughly 50 full native watcher starts (about 3.75 seconds at the measured per-watcher cost) before the 51st start triggers consolidation and collapses most of them into one recursive watcher on the shared parent. The early starts were wasted work.
5. A solution with fewer than 50 reference directories never consolidates at all, and pays the full per-directory cost for every one of them, with no ceiling related to how much consolidation would have saved.

### Minimal reproduction shape

This is close to the existing shape of `DefaultFileChangeWatcherTests`, so it should be easy to turn into a benchmark test in place:

1. Create N sibling directories, each containing one file to watch, for example `packages/package{i}/lib/package{i}.dll` for `i` in `0..N`.
2. Call `context.EnqueueWatchingFile` for each file in turn, timing wall-clock time.
3. For N below 50, expect close to N full native watcher starts and no consolidation.
4. For N above 50, expect consolidation to fire, but only after paying for the first 50 starts.
5. On macOS, expect wall-clock time to track N almost linearly until the cap, at roughly 45 to 75 ms per directory depending on hardware and OS build, then flatten once consolidation takes over.

### The fix I run today

I patched this locally and have been running it in production. Two small, separable changes:

1. A `consolidationThreshold` on `DefaultFileChangeWatcher` (I ship 8), separate from `_maxWatcherCount`. `FindBestNodeToConsolidateAndConsolidateIt_NoLock` also consolidates a node early, before the process-wide cap, once that single node's own active watcher count reaches the threshold. This bounds how many expensive native starts any one accumulating directory can cost, while solutions below the threshold keep exact, non-consolidated watching with no behaviour change. The threshold value is a judgement call, not a formal search; it may well want to be different, or platform-specific.
2. A `Directory.Exists` filter in `ReferenceFileChangeTracker` before registering the pre-declared reference directories, because a handful of declared directories (an SDK pack path, for example) did not exist on the machine and still paid a full watcher start for nothing.

Both changes carry tests: an identity test proving the same sequence of file-system events raises the exact same `FileChanged` callbacks with early consolidation on and off, threshold boundary tests, and a test confirming that further directories add no more watchers once a node is consolidated.

### Results with the fix applied

Five timed rounds per side on the same 96-directory fixture, a stock build carrying exactly the two changes against a stock unpatched build, both driven headless over stdio, same machine, idle:

- Warm solution load, median: 4131 ms unpatched, 1869 ms patched. 54.7 percent of the phase removed.
- The ramp flattens exactly as the reproduction section predicts. Unpatched, the load grows about 53 ms per reference directory up to the cap: 1736 / 2588 / 3857 / 4883 ms at 8 / 24 / 48 / 96 directories. Patched, it stays flat: 1762 / 1718 / 1839 / 1816 ms at the same points.
- Registration-attributable samples in the native captures fall from 430 to 46.
- Correctness held: the identity test proves the same `FileChanged` sequences with early consolidation on and off, and edit, add and delete detection latency stays at the same order.
- On this benchmark machine every declared reference directory existed, so the whole measured gain above belongs to the first change. The second change removes real cost only where declared directories are missing, as they were on the machine where I first traced this.

The native `sample` captures behind the before and after numbers are attached, together with the per-round timings. The thread-time percentages quoted earlier come from the managed traces, a separate instrument, and I can share those too.

I have not opened a pull request for this yet: the contributing guide asks for an issue and a maintainer's acknowledgement first. Once acknowledged, I will open two small PRs, one per change, and they are genuinely small:

1. **Early consolidation**: an 18-line change to `DefaultFileChangeWatcher` (a `consolidationThreshold` alongside the existing `_maxWatcherCount`, and one extra condition in `FindBestNodeToConsolidateAndConsolidateIt_NoLock` so a single accumulating node also consolidates when its own watcher count reaches the threshold). Around 160 lines of tests in `DefaultFileChangeWatcherTests`: the on/off identity test described above, threshold boundary cases, and a test that a consolidated node adds no native watchers for further directories. Solutions below the threshold keep today's behaviour exactly.
2. **Skip absent directories**: a one-line change to `ReferenceFileChangeTracker` (`Directory.Exists` before registering a pre-declared reference directory), with about 70 lines of tests covering declared-but-absent directories.

The two are independent; either can land without the other, and I am not attached to the threshold's default, its name, or whether it should be platform-specific.

### Related issues

- dotnet/roslyn#83101 introduced the consolidate-on-cap design this issue follows up. The fix here is a small addition to that design, not a reversal of it.
- dotnet/roslyn#82857 is the Linux inotify exhaustion problem dotnet/roslyn#83101 fixed. Different platform and different resource limit, but the same underlying pattern, and its discussion already sketched a threshold-based consolidation much like the one above.
- dotnet/runtime#77793, fixed by dotnet/runtime#121698, is the underlying per-call native start cost on macOS. It independently corroborates my per-watcher measurement, and its fix had not reached the SDK I tested against.
- zed-industries/zed#55746 is a third-party report of a related symptom: Roslyn's LSP watched-path registration causing watcher churn on the client side. Different code path, same underlying pressure.

[watcher-timings.zip](https://github.com/user-attachments/files/31199963/watcher-timings.zip)
[after.txt](https://github.com/user-attachments/files/31199961/after.txt)
[before.txt](https://github.com/user-attachments/files/31199962/before.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.