HarperFast / HarperFast/harper

Component file watcher can resolve its watch root to an ancestor directory, recursively scanning an unrelated tree

Open
#2,602 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A component's file watcher can resolve its watch root to an **ancestor** of the component
directory. When it does, chokidar recursively scans that entire unrelated tree, forever, on
both the main thread and every worker. Observed burning ~2.5 CPU cores per node, silently,
with nothing in the logs.

## Evidence

Found on a dev box where six Harper nodes were together consuming ~15 of 20 cores. Both the
main thread and the `http` worker were doing nothing but walking a directory tree:

```
worker "http" (95% busy): main thread (69% busy):
21% lstat 20% lstat
13% normalizePath chokidar 10% normalizeString node:path
11% normalizeString node:path 6% relative node:path
9% (garbage collector) 3% chokidar/handler.js:467
```

A conditional breakpoint on `readdirp`'s `_formatEntry` capturing `this._root` gave the scan
root directly:

```
SCAN ROOTS (entries walked in 4s):
4118 /home/kzyp/dev/tmp/agent-tmp <- TMPDIR, the shared scratch dir
```

That is one level *above* the node's own `ROOTPATH`
(`/home/kzyp/dev/tmp/agent-tmp/harper-integration-test-v9Slyi`). The size difference is the
whole problem:

| directory | files |
|---|---|
| the watch root that was actually scanned (`TMPDIR`) | **636,656** |
| the node's own `ROOTPATH` | **1** |

Only 3 inotify watches were held, and `_isIgnored` / `matchPatterns` were hot — so the walk
enumerated ~636k files and then discarded essentially all of them, repeatedly. The path-string
churn is what drives the GC time in the profile.

## Mechanism

`components/deriveCommonPatternBase.ts` reduces a component's pattern bases to their **common
path prefix**, with no clamp to the component directory. Two divergent bases collapse to a
shared ancestor:

```js
deriveCommonPatternBase(['/a/b/c', '/a/d']) // -> '/a'
```

`components/EntryHandler.ts:557-562` then passes an absolute base straight through to chokidar,
which by the comment right above it is deliberate — an absolute pattern "reaches the native
watch as spelled, bypassing the canonicalized `cwd`". So an ancestor-derived base becomes the
literal recursive watch root.

Nothing between those two points checks that the derived base is still inside
`component.directory`.

## Suggested fix

Clamp the derived base so it can never escape the component directory — if the common prefix
is not a descendant of `component.directory`, fall back to `component.directory` itself. A
watch root outside the component is never correct, so this seems safe to enforce as an
invariant rather than a heuristic.

## Affected versions

`components/deriveCommonPatternBase.ts` is present on `v5.1`, `v5.2`, and `main`.

## Confidence

The profile, the scan root, and the file-count asymmetry are **directly measured**. The link to
`deriveCommonPatternBase` is **inferred from code reading**: I killed the processes to reclaim
the machine before capturing the offending component's actual `patternBases`, and that is the
only code path I can find that produces an ancestor watch root. Worth confirming by logging
`watchPattern` in `EntryHandler` when it resolves outside `component.directory` — which is
probably worth a warning in its own right, given this was completely silent.

Contributor guide

Open the contributing guide

Research direction

Start with components/deriveCommonPatternBase.ts and then inspect components/EntryHandler.ts:557-562, where the derived absolute base is passed to chokidar. Confirm the watch root cannot resolve above component.directory, and verify that an affected component scans only its own directory rather than an ancestor tree.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.