monad-developers / monad-developers/ultrafuzz

Runtime suite is 2-63x slower per test on macOS than Linux; a full local run does not finish in 7 hours

Open
#1,060 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Summary

The runtime suite is dramatically slower on macOS than on Linux, on the same commit and the same tests. A full local run on main (8b985b90) was stopped after 7 hours without finishing; the Linux CI lanes for the same suite complete in 24-46 minutes each.

This only became visible after #1019 and #1025. Before those, most of these tests aborted in milliseconds on the symlinked os.tmpdir() (#1017), so the cost was hidden behind a failure. Nothing here is a regression from that work -- it is the bill arriving.

Per-test comparison, same commit

Darwin timings from the local run; Linux timings extracted from the CI runtime lanes on the same commit.

test darwin linux ratio
syncRun accepts the runner's correlation envelope and rejects a mismatched one 32.2 min 1.59 min 20x
syncRun accepts the pinned 0.35.0 usage payload and still bounds its new fields 29.8 min 13.76 min 2x
getRunHealth rejects every noncurrent status envelope without fallback or filtering 23.7 min 2.86 min 8x
refreshed controller rejects a drifted retained prompt snapshot 21.7 min 0.34 min 63x
syncRun binds an optional prerequisite digest before a final-boundary manifest swap 18.3 min 2.54 min 7x
syncRun succeeds when only an explicitly nonblocking specialist fails 17.7 min 0.61 min 29x

The slowest test on the Linux side of the whole sample is about a minute. On darwin, 147 tests exceed 60 seconds.

Aggregate

wall clock                 7h 00m, stopped before completion
measured test time        23.5 hours (tests run concurrently across files)
tests over 60s               147
slowest single test        32.2 min

It is not CPU

The machine was idle throughout. Sampled repeatedly during the run:

total CPU across test processes    7.6% - 18.8%

An M-series laptop at ~90% idle taking 63x a CI container points at IO or syscall behaviour rather than compute. During the run the system temporary directory held 378,807 live files across 149 roots, which is a plausible contributor but has not been isolated.

What this is not

  • Not the leak in #1018. Those files are live fixtures belonging to running tests, not abandoned state; the temporary directory was cleared immediately before this run.
  • Not sharding. Linux CI splits the suite across lanes, but the numbers above are per-test on both sides, so parallelism does not explain them.
  • Not a regression from #1019 / #1025. Those made the tests execute rather than abort; they did not make anything slower per unit of work.

Why it matters

A contributor on macOS cannot run the runtime suite as part of normal work. Seven hours without finishing means the practical options are to push and let CI answer, or to run a narrow selection and hope the rest is unaffected. Both weaken the suite as a local safety net, and it is the platform the project now supports for campaigns (#1011, #1016).

Not diagnosed

I have not isolated the cause and am not proposing a fix here. The obvious next step is to profile one of the extreme cases -- refreshed controller rejects a drifted retained prompt snapshot at 63x is the cleanest candidate, since it is only 20 seconds on Linux -- and determine whether the time is in filesystem syscalls, process spawning, or something else specific to darwin.

Environment

  • macOS (Darwin 25.6.0) arm64, node v26.3.1, pnpm 11.1.1
  • ultrafuzz main @ 8b985b90
  • Linux figures from the CI runtime lanes on the same commit
  • Default TMPDIR; temporary directory cleared before the run

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by profiling the refreshed controller rejects a drifted retained prompt snapshot test on macOS and compare its filesystem syscalls, process spawning, and other runtime costs with Linux. The issue is done when the platform-specific cause of the extreme slowdown is isolated and a concrete remediation path is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, node.js, typescript
Domain
operating-systems, performance, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.