Root stuck in MustScanSubDirs KernelDropped recrawl loop never backs off (14,459 recrawls, done_initial never true)
- Dominant language
- C++
- Stars
- 13.7k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
A root that enters a `MustScanSubDirs KernelDropped` recrawl loop recrawls forever. There is no backoff, no rate limit, no retry cap, no give-up threshold, and no error surfaced to anyone — it just consumes a core indefinitely.
One root on my machine recrawled **14,459 times in ~19 minutes** (~13/sec) with `"done_initial": false` the entire time. It never once completed its initial crawl. Three such roots pinned all 10 cores of an M1 Max at load average 254.
## Environment
| | |
|---|---|
| watchman | `2026.01.05.00` (Homebrew) |
| macOS | 26.5.2 (build 25F84) |
| Hardware | MacBookPro18,4, Apple M1 Max |
| Filesystem | APFS, case-insensitive |
| Watcher | fsevents (auto) |
Roots were ordinary TypeScript project directories, ~7,700 files each. Not monorepos.
## Evidence
`watchman debug-status`:
```json
{
"path": "/Users//",
"uptime": 1133,
"recrawl_info": {
"count": 14460,
"should-recrawl": true,
"completed": null,
"started": -59,
"reason": "MustScanSubDirs KernelDropped"
},
"watcher": "fsevents",
"done_initial": false,
"queries": [],
"cancelled": false,
"crawl-status": "re-crawling for 59 ms"
}
```
`count: 14460` against `uptime: 1133` seconds. `done_initial: false`. `queries: []` — no client was even connected.
The log is this, flat, forever:
```
[notify ] MustScanSubDirs KernelDropped: scheduling a tree recrawl
[io ] PERF: {... "description": "full-crawl"}
[io ] recrawl complete
[notify ] MustScanSubDirs KernelDropped: scheduling a tree recrawl
```
The interval never grows. Recrawl 14,000 is scheduled exactly as eagerly as recrawl 1.
While this ran: `fseventsd` 164%, `watchman` 103%, `filecoordinationd` 53%, plus unrelated software that merely consumes file events (a password manager, an endpoint-security extension) dragged to 100%+. Load average 254, 122 runnable processes. After `watch-del-all && shutdown-server`: load 11.7, 6 runnable.
The Homebrew LaunchAgent ships `Nice=-5`, so this is scheduled *above* normal user processes.
## The Bug
To be clear which layer I am reporting: the kernel dropping events is **not** a watchman bug. `KernelDropped` means the events were lost before watchman saw them, and recrawling to resync is the correct response.
The defect is that the recovery mechanism has no limiting behaviour of any kind:
- **No backoff.** Flat interval across 14,459 consecutive failures.
- **No cap.** Nothing stops at 10, 100, or 10,000 attempts.
- **No health state.** A root failing every recrawl is treated identically to a healthy one.
- **No signal.** The warning string in `debug-status` is the only indication, and you have to know to go looking. Nothing is logged at error level, no query fails, nothing tells the user their machine is being eaten (apart from heat and the fans taking off like a jumbo jet).
A root that has attempted 14,459 recrawls without once completing its initial crawl is doing **zero useful work** while consuming a full core. That state is detectable in a single comparison — `recrawl_count` high, `done_initial` still false — and nothing acts on it.
The [troubleshooting docs](https://facebook.github.io/watchman/docs/troubleshooting) call frequent recrawls "undesirable" and stop there. No circuit breaker exists.
## Suggested Directions
1. **Exponential backoff on consecutive failed recrawls** — if N recrawls complete without `done_initial` becoming true, grow the delay instead of immediately rescheduling.
2. **A give-up threshold** — mark the root unhealthy, stop recrawling until a client explicitly re-establishes the watch, and return the error in query responses.
3. **Log at error level** once a root crosses some recrawl count, so this is discoverable without running `debug-status`.
4. **Distinguish `KernelDropped` from `UserDropped` in the warning text.** Different causes, different remedies — but the current message sends everyone to `fsevents_latency`, which cannot help `KernelDropped`.
## Reproduction
I do not have a minimal reproduction; the kernel-side drop depends on machine-wide FSEvents pressure I have not isolated.
Most promising lead: facebook/metro#814 ("Metro seems to leave orphaned watchman watches frequently"). That issue is about clients failing to release watches and says nothing about `KernelDropped` — I am not claiming it causes the drops. But orphaned roots plausibly supply the precondition: client-less streams nobody monitors, restored at every boot from the statefile. Accumulate several via normal jest/metro use, apply FSEvents pressure, watch `recrawl_count`.
Note that regardless of what triggers the first drop, the unbounded retry is independently fixable.
Happy to run diagnostics or supply full logs. This has recurred on three unrelated projects on this machine, so I can likely reproduce on request.
## Possibly Related
- #1157, #981 — same loop shape, but `UserDropped` rather than `KernelDropped`
- #1306 — different failure (process explosion), also macOS
- facebook/metro#814 — orphaned watches; see Reproduction
Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.