docs(pm): benchmark reference for cold and warm installs

Start with the documented pm-bench-all, pm-bench-phases, and bench/pm-bench.sh harnesses, then review the linked workflow evidence and existing follow-ups. Done means the legacy harness is retired or clearly deprecated for performance claims and a Linux-only manual-dispatch option is documented or added without changing the benchmark contract.

Purpose

Keep a stable, linkable reference for utoo package-manager benchmark results and methodology. Future PRs should link this issue instead of quoting results from the legacy pm-bench-all harness.

Benchmark contract

Compare these three user-visible states:

1. Full cold (p0): no lockfile, no manifest cache, no tarball/package store, no node_modules.
2. Lock-only cold (p3): lockfile exists, but tarball/package store and node_modules are removed.
3. Warm (p4): lockfile and tarball/package store are populated; only node_modules is removed.

Do not use “no lockfile but warm manifest cache”. Bun keeps manifests fresh for five minutes without revalidation, so that state disproportionately benefits Bun and is not one of the intended scenarios.

The phase harness uses explicit per-PM stores, restores each PM's own lockfile, disables lifecycle scripts, alternates PM order between rounds, and records network traffic so an accidentally cold “warm” cell can be rejected.

Results

Results are grouped by date → registry → execution environment. Runs from different dates, registries, or environments must not be mixed statistically.

2026-08-27

registry.npmjs.org · GitHub Actions

• Harness branch/commit: codex/pm-bench-pnpm12 / 2204fd0f4
• Project: ant-design
• Registry: https://registry.npmjs.org
• Workflow: https://github.com/utooland/utoo/actions/runs/33039114486
• Versions: utoo built from 2204fd0f4, Bun 1.4.0, pnpm 10.34.5, pnpm 12.0.0, aube 1.29.1
• Sampling: one measured round for p0/p3; 5 interleaved rounds for p1/p4. Treat p0/p3 as smoke evidence only; the repeated Linux China run below is the measured comparison.

Linux (ubuntu-latest)

Mode	utoo	Bun	pnpm 10	pnpm 12	aube
Full cold (p0)	7.87s	9.62s	29.56s	10.14s	16.13s
Resolve only (p1)	2.12s ± 0.31s	1.94s ± 0.06s	9.93s ± 0.08s	0.95s ± 0.02s	4.89s ± 0.22s
Lock-only cold (p3)	9.66s	10.40s	18.32s	10.58s	—
Warm (p4)	2.35s ± 0.04s	3.65s ± 0.34s	6.14s ± 0.07s	3.39s ± 0.24s	—

macOS (macos-latest, Apple Silicon)

Mode	utoo	Bun	pnpm 10	pnpm 12	aube
Full cold (p0)	20.46s	23.42s	57.19s	30.32s	51.44s
Resolve only (p1)	3.16s ± 0.40s	2.12s ± 0.13s	11.07s ± 0.80s	1.67s ± 0.45s	5.35s ± 0.21s
Lock-only cold (p3)	15.80s	25.31s	60.12s	31.55s	—
Warm (p4)	4.76s ± 0.20s	6.43s ± 0.42s	24.69s ± 3.67s	21.89s ± 2.63s	—

In this smoke run, pnpm 12 was fastest for the synthetic resolve-only phase, while utoo was fastest for all three user-visible install states on both platforms. Aube completed p0/p1, but its p3/p4 cells were rejected because the seed step did not produce the expected aube-lock.yaml.

registry.npmmirror.com · Linux China

• Harness branch/commit: codex/pm-bench-pnpm12 / 2204fd0f4
• Project: ant-design at e8fbc23658a323812864d56d53b5d25b73784dd1
• Host: Linux x86_64 (China), 96 logical CPUs, 16 GiB RAM, Node.js 22.18.0
• Registry: https://registry.npmmirror.com
• Network: direct connection; proxy environment variables were unset and the benchmark command explicitly selected npmmirror
• Versions: utoo built from 2204fd0f4, Bun 1.3.14, pnpm 10.34.5, pnpm 12.0.0, aube 1.29.1
• Sampling: 3 interleaved rounds for p0/p3; 5 for p4

Mode	utoo	Bun	pnpm 10	pnpm 12	aube
Full cold (p0)	12.04s ± 1.25s	19.63s ± 1.32s	33.88s ± 0.64s	18.50s ± 6.22s	26.47s ± 7.65s
Lock-only cold (p3)	22.28s ± 12.71s	31.75s ± 15.11s	40.70s ± 11.64s	31.67s ± 10.09s	—
Warm (p4)	1.22s ± 0.07s	4.58s ± 0.32s	9.86s ± 0.48s	2.60s ± 0.09s	—

Relative to utoo's mean, pnpm 12 was 54% slower in p0, 42% slower in p3, and 113% slower in p4. The p0/p3 variance is high enough that these two percentages should be treated as directional; the low-variance p4 result is much stronger. The warm cells retained each PM's lockfile and store while transferring only ~270 B RX for utoo and ~2 KB RX for pnpm 12, confirming that p4 was genuinely warm.

Aube 1.29.1 completed p0, but its --lockfile-only command did not produce the expected aube-lock.yaml. The harness therefore rejected its p3/p4 cells instead of timing stale or lockless state.

Raw utoo vs pnpm 12 wall-time samples

• p0 utoo: 11.87s, 10.89s, 13.37s
• p0 pnpm 12: 25.51s, 16.38s, 13.62s
• p3 utoo: 9.50s, 34.91s, 22.42s
• p3 pnpm 12: 41.85s, 31.49s, 21.68s
• p4 utoo: 1.16s, 1.14s, 1.24s, 1.32s, 1.25s
• p4 pnpm 12: 2.66s, 2.64s, 2.64s, 2.45s, 2.59s

2026-07-21

registry.npmjs.org · GitHub Actions

• Harness branch/commit: codex/aube-phase-benchmark / 7a76aa3
• Project: ant-design
• Registry: https://registry.npmjs.org
• Workflow: https://github.com/utooland/utoo/actions/runs/29797404521
• Sampling: 3 interleaved rounds for p0/p3; 5 for p4
• aube: 1.29.1 with trustPolicy: off

Linux (ubuntu-latest)

Mode	utoo	Bun	aube
Full cold (p0)	8.13s ± 0.33s	9.70s ± 0.24s	17.54s ± 0.74s
Lock-only cold (p3)	7.22s ± 1.24s	7.52s ± 0.04s	8.37s ± 1.04s
Warm (p4)	2.25s ± 0.12s	3.58s ± 0.05s	2.59s ± 0.13s

macOS (macos-latest, Apple Silicon)

Mode	utoo	Bun	aube
Full cold (p0)	19.19s ± 2.43s	22.81s ± 1.29s	55.28s ± 5.78s
Lock-only cold (p3)	12.03s ± 0.53s	15.58s ± 3.94s	31.31s ± 3.12s
Warm (p4)	3.81s ± 0.19s	5.37s ± 0.22s	14.76s ± 0.26s

The pnpm cells from this workflow are intentionally omitted: ant-design's .npmrc contains package-lock=false, which pnpm 10 mapped to lockfile=false, so those cells had no pnpm-lock.yaml and were not valid p3/p4 measurements. Harness commit f5210b24 now forces --config.lockfile=true.

Corrected pnpm 10 validation

• Hardened harness commit: 1e88fa2b
• Workflow: https://github.com/utooland/utoo/actions/runs/29809892297

Platform	Full cold (p0)	Lock-only cold (p3)	Warm (p4)
Linux	26.20s ± 0.81s	18.01s ± 1.03s	7.19s ± 0.02s
macOS	70.58s ± 13.74s	57.25s ± 15.07s	31.88s ± 2.58s

The Linux warm cell retained a ~1 MB pnpm-lock.yaml and ~1.45 GB store while transferring only ~7 KB RX / 18 KB TX. This confirms the corrected p4 cell is genuinely warm. The same run rejected aube p3/p4 because the expected seeded lockfile was absent rather than timing stale state.

registry.npmmirror.com · Linux China

• Host: Linux x86_64 (China)
• Registry: https://registry.npmmirror.com
• Network: direct connection; no HTTP_PROXY, HTTPS_PROXY, NO_PROXY, npm proxy, macmini host entry, or macmini shell configuration was present
• Versions: utoo 1.1.3, Bun 1.3.14, pnpm 10.34.5, aube 1.29.1
• Sampling: 3 interleaved rounds for p0/p3; 5 for p4

Mode	utoo	Bun	pnpm 10	aube
Full cold (p0)	7.84s ± 2.17s	13.21s ± 2.36s	39.32s ± 16.56s	27.63s ± 6.47s
Lock-only cold (p3)	6.64s ± 1.64s	8.62s ± 3.57s	15.14s ± 0.77s	6.42s ± 0.31s
Warm (p4)	1.09s ± 0.05s	2.72s ± 0.05s	7.89s ± 0.38s	1.60s ± 0.07s

For the corrected pnpm warm run, network RX was ~4 KB and the lockfile/store were present, confirming it is a true warm measurement. The p3 utoo/aube cell was repeated for 9 paired, order-alternating rounds after hardening the harness to reject missing locks and failed prepares. Its paired utoo-minus-aube delta was +0.214s with SE 0.506s (t=0.42), so the small mean difference is noise: treat utoo and aube as tied for p3. Utoo is clearly fastest for p0 and p4.

Historical confirmation

PR #3199 showed the same utoo-vs-Bun direction under the phase-isolated methodology: Linux p0 7.71s vs 9.51s, Linux p4 2.40s vs 3.68s, and macOS p4 3.76s vs 4.41s.

Reference: https://github.com/utooland/utoo/pull/3199

Invalid legacy result · 2026-07-20 · registry.npmjs.org

Run https://github.com/utooland/utoo/actions/runs/29740213943 used the older bench/pm-bench.sh harness. Its “warm” preparation runs git clean -dfx, deleting the generated lockfile while retaining global caches. It therefore measures fresh resolution with a warm cache and exposes Bun's five-minute manifest fast path. It also uses shared/default cache locations and does not pair/interleave rounds as strictly as pm-bench-phases.sh.

Do not cite that run as a utoo-vs-Bun comparison.

Aube failure diagnosis

The legacy run's aube 1.29.1 cells failed with ERR_AUBE_TRUST_DOWNGRADE: aube's default no-downgrade supply-chain policy rejected @swc/core@1.9.2. This is a policy rejection, not a timing failure. The phase harness synthesizes:

trustPolicy: off

That permits all PMs to install the same dependency graph and produced the valid aube results above.

Follow-ups

• Retire or clearly deprecate pm-bench-all for performance claims; use pm-bench-phases for PR evidence.
• Add aube with explicit XDG cache/store isolation, lock restoration, and trustPolicy: off.
• Add pnpm 10 and pnpm 12 with separate explicit stores and force lockfile generation even when an npm project sets package-lock=false.
• Capture failed command output and skip invalid JSON cells instead of crashing the summary.
• Reject missing seeded lockfiles and abort on prepare failures so stale stores cannot masquerade as cold installs (1e88fa2b).
• Add a Linux-only manual-dispatch option so registry benchmarks do not need to consume a macOS runner.