HarperFast / HarperFast/harper
package_component still walks the install through a loaded component's node_modules/harperdb, and skipping the path breaks fresh deploys
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
## Summary
A legacy component that has been **loaded** still makes `package_component` walk the entire Harper install, through `node_modules/harperdb`. This is the residue of #2487, which fixed the same problem for `node_modules/harper`.
Skipping the path is **not** the fix, and that is what makes this its own issue.
## Why the obvious fix is wrong
`symlinkHarperModule` treats the two names differently (`components/componentLoader.ts`):
- `harper` — created unconditionally: `rmSync(harperModule, { recursive: true, force: true })` then a symlink to `PACKAGE_ROOT`, whether or not the path existed.
- `harperdb` — only **repaired when the path already exists** (`harperdbModulePresent && !harperdbModuleLinked`).
So the sequence that breaks is:
1. A legacy component ships a real npm-installed `node_modules/harperdb`.
2. It is loaded, so the loader replaces that directory with a symlink to the install.
3. `package_component` packages it. With the link excluded, the archive contains `node_modules/` but **no** `harperdb`.
4. The archive is deployed to a **fresh** node. `installApplication` returns early because `node_modules` exists (`components/Application.ts`, "already has node_modules; skipping install"), so nothing installs `harperdb`.
5. On load, the loader's present-gate does not fire, because the path does not exist. Nothing recreates it.
6. A native-mode component importing `harperdb` fails. `jsLoader`'s synthetic alias (`security/jsLoader.ts`) only covers VM-loader modes, so bundled output or a natively-resolved dependency is not covered.
Both available choices are therefore wrong on their own:
| Treatment | Failure |
|---|---|
| Don't exclude it (today) | Packaging a loaded legacy component packages the whole install — the #2487 symptom, ~700MB and `Maximum response size reached` |
| Exclude it (by name or by resolved identity) | Fresh target loses `harperdb` with no recovery path; native `import from 'harperdb'` fails |
An earlier revision of the #2487 branch implemented the identity-based exclusion and introduced exactly the second failure; it was backed out rather than shipped.
## Suggested fix
Pack a **placeholder** at `node_modules/harperdb` rather than skipping it — an empty directory entry is enough, because the loader's gate is presence, not content. The target then extracts a path that exists, the present-gate fires on the next load, and the loader repairs it into the install link. That preserves both properties: the archive does not carry the install, and the target can still resolve `harperdb`.
Mechanically this needs an injected tar entry rather than an `ignore` predicate — `tar-fs`'s `ignore` can only drop entries and `map` can only rewrite headers, so it likely means `finalize: false` plus an explicit `pack.entry()` for the directory, and a matching adjustment in `scanPackageDirectory` so the size estimate still agrees with the archive.
Worth confirming first whether any supported component still imports `harperdb` natively; if the answer is no, excluding it outright becomes acceptable and the fix is a two-line change instead.
## Verification route
Extend `unitTests/components/packageComponent.test.js`: a fixture whose `node_modules/harperdb` is a symlink to `PACKAGE_ROOT` must produce an archive that (a) contains no install content under that path and (b) still contains the path itself, so a fresh extraction has something for the loader's present-gate to find. An integration test that deploys a legacy-shaped component with `restart: true`, packages it, and redeploys the payload to a second instance would cover the end-to-end sequence above.
## Provenance
Split out of #2487 (PR pending) while fixing the `harper` half. Found by cross-model review of that branch, which caught the fresh-deployment regression the identity-based exclusion introduced.
Contributor guide
Research direction
Read components/componentLoader.ts, components/Application.ts, security/jsLoader.ts, and the packageComponent implementation first, then run unitTests/components/packageComponent.test.js. Confirm whether supported components import harperdb natively and trace how packaging and scanPackageDirectory estimate the archive. Done means the archive contains the node_modules/harperdb path without install contents, and the fresh-deployment behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100