HarperFast / HarperFast/harper

package_component still walks the install through a loaded component's node_modules/harperdb, and skipping the path breaks fresh deploys

Open
#2,577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## Summary

A legacy component that has been **loaded** still makes `package_component` walk the entire Harper install, through `node_modules/harperdb`. This is the residue of #2487, which fixed the same problem for `node_modules/harper`.

Skipping the path is **not** the fix, and that is what makes this its own issue.

## Why the obvious fix is wrong

`symlinkHarperModule` treats the two names differently (`components/componentLoader.ts`):

- `harper` — created unconditionally: `rmSync(harperModule, { recursive: true, force: true })` then a symlink to `PACKAGE_ROOT`, whether or not the path existed.
- `harperdb` — only **repaired when the path already exists** (`harperdbModulePresent && !harperdbModuleLinked`).

So the sequence that breaks is:

1. A legacy component ships a real npm-installed `node_modules/harperdb`.
2. It is loaded, so the loader replaces that directory with a symlink to the install.
3. `package_component` packages it. With the link excluded, the archive contains `node_modules/` but **no** `harperdb`.
4. The archive is deployed to a **fresh** node. `installApplication` returns early because `node_modules` exists (`components/Application.ts`, "already has node_modules; skipping install"), so nothing installs `harperdb`.
5. On load, the loader's present-gate does not fire, because the path does not exist. Nothing recreates it.
6. A native-mode component importing `harperdb` fails. `jsLoader`'s synthetic alias (`security/jsLoader.ts`) only covers VM-loader modes, so bundled output or a natively-resolved dependency is not covered.

Both available choices are therefore wrong on their own:

| Treatment | Failure |
|---|---|
| Don't exclude it (today) | Packaging a loaded legacy component packages the whole install — the #2487 symptom, ~700MB and `Maximum response size reached` |
| Exclude it (by name or by resolved identity) | Fresh target loses `harperdb` with no recovery path; native `import from 'harperdb'` fails |

An earlier revision of the #2487 branch implemented the identity-based exclusion and introduced exactly the second failure; it was backed out rather than shipped.

## Suggested fix

Pack a **placeholder** at `node_modules/harperdb` rather than skipping it — an empty directory entry is enough, because the loader's gate is presence, not content. The target then extracts a path that exists, the present-gate fires on the next load, and the loader repairs it into the install link. That preserves both properties: the archive does not carry the install, and the target can still resolve `harperdb`.

Mechanically this needs an injected tar entry rather than an `ignore` predicate — `tar-fs`'s `ignore` can only drop entries and `map` can only rewrite headers, so it likely means `finalize: false` plus an explicit `pack.entry()` for the directory, and a matching adjustment in `scanPackageDirectory` so the size estimate still agrees with the archive.

Worth confirming first whether any supported component still imports `harperdb` natively; if the answer is no, excluding it outright becomes acceptable and the fix is a two-line change instead.

## Verification route

Extend `unitTests/components/packageComponent.test.js`: a fixture whose `node_modules/harperdb` is a symlink to `PACKAGE_ROOT` must produce an archive that (a) contains no install content under that path and (b) still contains the path itself, so a fresh extraction has something for the loader's present-gate to find. An integration test that deploys a legacy-shaped component with `restart: true`, packages it, and redeploys the payload to a second instance would cover the end-to-end sequence above.

## Provenance

Split out of #2487 (PR pending) while fixing the `harper` half. Found by cross-model review of that branch, which caught the fresh-deployment regression the identity-based exclusion introduced.

Contributor guide

Open the contributing guide

Research direction

Read components/componentLoader.ts, components/Application.ts, security/jsLoader.ts, and the packageComponent implementation first, then run unitTests/components/packageComponent.test.js. Confirm whether supported components import harperdb natively and trace how packaging and scanPackageDirectory estimate the archive. Done means the archive contains the node_modules/harperdb path without install contents, and the fresh-deployment behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.