HarperFast / HarperFast/harper
Reduce Harper's install footprint: 416MB, ~230MB of it unusable on any given install
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`npm install harper@5.2.0` (production only) lands **416MB**. Measured breakdown:
| | size | |
|---|---|---|
| `@harperfast/rocksdb-js-*` | 94M | **8 platform prebuilds — all installed** |
| `harper/studio` | 95M | bundled UI |
| `@lmdb/lmdb-*` | 38M | **7 platform prebuilds — all installed** |
| mathjs · rxjs · @fastify · @aws-sdk | 52M | |
| alasql · fastify · @babel · @smithy · moment | 40M | |
| `harper/dist` | 13M | the server code |
On a `darwin/arm64` host, **13 of the 15 native prebuilds can never load** — `rocksdb-js-win32-x64` (7M), `lmdb-linux-arm64` (16M), and so on. That is ~132MB, 32% of the install, dead weight on every machine regardless of platform. They are real binaries, not stubs: `file` reports `rocksdb-js-win32-x64/rocksdb-js.node` as `PE32+ executable (DLL) x86-64, for MS Windows`, sitting next to the one Mach-O arm64 bundle this host can actually load.
For scale: the react-native tree removed in #1937 was ~140MB. This is the same size again.
**Read the two sections below before costing any of this** — the prebuild half only affects npm ≤ 11 and npm 12 already resolves it, which makes **studio (95MB, 23%) the durable lever** and the only one that holds across both npm majors.
## Why the prebuilds aren't filtered
npm's `os`/`cpu` filtering works — it just never runs for us.
```
# fresh resolve, no shrinkwrap involved
$ npm install @harperfast/rocksdb-js@2.5.0
rocksdb-js rocksdb-js-darwin-arm64 <- 1 prebuild, correct
# via harper, which ships npm-shrinkwrap.json
$ npm install harper@5.2.0
rocksdb-js-darwin-arm64 rocksdb-js-darwin-x64
rocksdb-js-linux-arm64-glibc rocksdb-js-linux-arm64-musl
rocksdb-js-linux-x64-glibc rocksdb-js-linux-x64-musl
rocksdb-js-win32-arm64 rocksdb-js-win32-x64 <- all 8
```
The metadata is not the problem. Every entry carries `optional: true` plus correct `os`/`cpu`, in both the published shrinkwrap and the consumer's generated lockfile, and the packages themselves declare `os`/`cpu` correctly.
**Correction to an earlier version of this issue,** which claimed a bundled shrinkwrap skips `os`/`cpu` filtering outright. That is wrong. Four measurements on the same host pin down the real discriminator:
| shape | prebuilds |
|---|---|
| platform packages as **root** `optionalDependencies`, `npm ci` from a lockfile pinning all 8 | **1** ✓ |
| `@harperfast/rocksdb-js` as a root dependency, fresh resolve | **1** ✓ |
| harper as the **root** project (extracted tarball, install in place) | **1** ✓ |
| harper as a **nested dependency** (`npm install harper`) | **8** ✗ |
So npm *does* re-apply platform filtering when reifying from a lockfile — it just doesn't re-filter a shrinkwrapped **dependency's** nested tree. Root versus nested is the discriminator, not the presence of a lockfile.
That rules out the cheap fix: moving the declaration up to the parent does not help. Verified directly by serving a package that declares all 8 prebuilds as its *own* `optionalDependencies` with a shrinkwrap pinning all 8, installed as a dependency through the registry code path — all 8 still land. Nothing in harper's own `package.json` can change consumer behaviour here.
**And npm 12 resolves this half for free.** It removed `npm-shrinkwrap.json` support entirely, so consumers resolve fresh and filtering applies: 1 prebuild each for rocksdb-js and lmdb. It also brings back the react-native tree and voids version pinning, for a net-larger install — tracked in #2172. Which means the ~132MB is a **npm ≤ 11 problem with a known expiry**, and any fix costed here should be weighed against simply aging out.
## The two install paths have opposite pathologies
Same package, two ways in, each broken differently:
| | registry `npm install harper` | Docker (extract + `npm install` in place, since #2042) |
|---|---|---|
| rocksdb prebuilds | **8** | 1 ✓ |
| lmdb prebuilds | **7** | 1 ✓ |
| react-native tree | absent ✓ | **present (~144MB)** — #2043 |
| prod tree | 416M | 338M |
Reading harper as a nested shrinkwrapped dependency preserves the react-native prune but skips platform filtering. Reifying it as the root project restores filtering but re-adds alasql's optional `react-native-fs` edge. Both are branches of the same question: *is harper the root, or a shrinkwrapped dependency?*
Worth stating plainly: **a single published shrinkwrap cannot be platform-specific**, so "prune the wrong platforms from the shrinkwrap" is not available — it has to serve every consumer.
## Levers, roughly in order of size
1. **Platform prebuilds (~132MB, registry installs on npm ≤ 11).** Ranked first by size, but **probably not worth building for**: it only affects npm ≤ 11, npm 12 already fixes it, and the fixes available to us are all unattractive. Declaring on the parent does not work (measured above). A post-install prune would have to run on the consumer's machine, which we do not control. Moving prebuilds to a runtime fetch is a real design change to `@harperfast/rocksdb-js` and its owner's call. Dropping the shrinkwrap forfeits the pinning #2042 just landed. Reporting upstream is also moot — npm already removed the feature, which is exactly how this behaviour goes away. Recommend tracking it and letting npm 12 adoption close it, unless someone wants the win sooner on npm 11.
2. **Studio (95MB, 23% — both install paths, both npm majors).** Given the above, this is effectively **lever #1**: the largest share that does not age out, and the only one unaffected by which npm a consumer runs. Every install carries the bundled UI whether or not it is ever served. Making it optional or fetched on demand is the single largest one-decision win, but that is a devexp/product call rather than a packaging one, so flagging it for whoever owns Studio rather than proposing a direction. @dawsontoth
3. **react-native subtree in the image (~144MB).** Already tracked in #2043.
4. **Heavy dependency tail.** `mathjs` (17M), `rxjs` (12M), `moment` (6M, deprecated upstream), `lodash` (5M). Each needs actual-usage analysis before anything is proposed — `dependencies.md` sets that bar — and some may be lightly used enough to replace or defer-load.
## Not covered here
Runtime footprint — resident memory, boot-time module graph, startup latency — is a separate axis from bytes on disk and is not measured above. Worth its own issue if there is appetite.
## Method
All numbers from `npm install --omit=dev` of `harper@5.2.0` on darwin/arm64, and from extracting the published tarball and installing in place to reproduce the Docker path. Happy to re-run on linux/x64 if the platform split matters for prioritisation.
`--ignore-scripts` was used for most runs and does not affect any of it: `@harperfast/rocksdb-js` declares no `install`/`preinstall`/`postinstall` hook — every script in its manifest is dev-facing — and re-running with scripts enabled gave the same counts and sizes (8 prebuilds, 93M, 415M vs 8 / 94M / 416M). Worth stating because a postinstall prune would have been the obvious explanation for these numbers being an artifact, and it isn't one.
The npm 12 comparison used `npx npm@12.0.2`. Bundled-npm versions per Node line were read from nodejs.org release metadata.
## References
- #1937 — react-native-fs shrinkwrap prune (the ~140MB precedent)
- #2042 — Docker install pinning fix
- #2043 — react-native subtree still in the image
- #2172 — npm 12 removes shrinkwrap support; resolves the prebuild half and voids the pinning
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Begin with npm-shrinkwrap.json, package.json, and the production install measurements described in the issue; reproduce npm install --omit=dev for the registry and Docker paths with the stated npm versions. Before changing anything, establish which lever is selected; done means an agreed scope and a measured reduction without regressing the referenced install paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, javascript, nodejs
- Domain
- build-system, devops, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100