monad-developers / monad-developers/ultrafuzz
Runtime suite is 2-63x slower per test on macOS than Linux; a full local run does not finish in 7 hours
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Summary
The runtime suite is dramatically slower on macOS than on Linux, on the same commit and the same tests. A full local run on main (8b985b90) was stopped after 7 hours without finishing; the Linux CI lanes for the same suite complete in 24-46 minutes each.
This only became visible after #1019 and #1025. Before those, most of these tests aborted in milliseconds on the symlinked os.tmpdir() (#1017), so the cost was hidden behind a failure. Nothing here is a regression from that work -- it is the bill arriving.
Per-test comparison, same commit
Darwin timings from the local run; Linux timings extracted from the CI runtime lanes on the same commit.
| test | darwin | linux | ratio |
|---|---|---|---|
syncRun accepts the runner's correlation envelope and rejects a mismatched one |
32.2 min | 1.59 min | 20x |
syncRun accepts the pinned 0.35.0 usage payload and still bounds its new fields |
29.8 min | 13.76 min | 2x |
getRunHealth rejects every noncurrent status envelope without fallback or filtering |
23.7 min | 2.86 min | 8x |
refreshed controller rejects a drifted retained prompt snapshot |
21.7 min | 0.34 min | 63x |
syncRun binds an optional prerequisite digest before a final-boundary manifest swap |
18.3 min | 2.54 min | 7x |
syncRun succeeds when only an explicitly nonblocking specialist fails |
17.7 min | 0.61 min | 29x |
The slowest test on the Linux side of the whole sample is about a minute. On darwin, 147 tests exceed 60 seconds.
Aggregate
wall clock 7h 00m, stopped before completion
measured test time 23.5 hours (tests run concurrently across files)
tests over 60s 147
slowest single test 32.2 min
It is not CPU
The machine was idle throughout. Sampled repeatedly during the run:
total CPU across test processes 7.6% - 18.8%
An M-series laptop at ~90% idle taking 63x a CI container points at IO or syscall behaviour rather than compute. During the run the system temporary directory held 378,807 live files across 149 roots, which is a plausible contributor but has not been isolated.
What this is not
- Not the leak in #1018. Those files are live fixtures belonging to running tests, not abandoned state; the temporary directory was cleared immediately before this run.
- Not sharding. Linux CI splits the suite across lanes, but the numbers above are per-test on both sides, so parallelism does not explain them.
- Not a regression from #1019 / #1025. Those made the tests execute rather than abort; they did not make anything slower per unit of work.
Why it matters
A contributor on macOS cannot run the runtime suite as part of normal work. Seven hours without finishing means the practical options are to push and let CI answer, or to run a narrow selection and hope the rest is unaffected. Both weaken the suite as a local safety net, and it is the platform the project now supports for campaigns (#1011, #1016).
Not diagnosed
I have not isolated the cause and am not proposing a fix here. The obvious next step is to profile one of the extreme cases -- refreshed controller rejects a drifted retained prompt snapshot at 63x is the cleanest candidate, since it is only 20 seconds on Linux -- and determine whether the time is in filesystem syscalls, process spawning, or something else specific to darwin.
Environment
- macOS (Darwin 25.6.0) arm64, node v26.3.1, pnpm 11.1.1
- ultrafuzz
main@8b985b90 - Linux figures from the CI runtime lanes on the same commit
- Default
TMPDIR; temporary directory cleared before the run
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling the refreshed controller rejects a drifted retained prompt snapshot test on macOS and compare its filesystem syscalls, process spawning, and other runtime costs with Linux. The issue is done when the platform-specific cause of the extreme slowdown is isolated and a concrete remediation path is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, node.js, typescript
- Domain
- operating-systems, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100