HarperFast / HarperFast/harper-pro
Nightly Large-Data Stress workflow reds on transient npm-registry 5xx during bin relink
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
The `Large-Data Stress Tests` nightly (`.github/workflows/stress-large-data.yaml`) failed on 2026-07-29 without running a single test. The failure was in the `Relink bin scripts` step of the `large-clone` job, which runs `npm install --ignore-scripts` against the artifact-restored `node_modules/` purely to recreate `node_modules/.bin` symlinks after the build artifact is downloaded.
```
npm error code E503
npm error 503 Service Unavailable - GET https://registry.npmjs.org/@babel%2fparser - Service Unavailable
##[error]Process completed with exit code 1.
```
## Evidence it was a registry blip, not a runner/network problem
- The sibling matrix job in the same run, `large-catchup` (same self-hosted runner pool, same build artifact, same commit), completed successfully in 29m3s.
- The two prior nightlies (2026-07-27, 2026-07-28) passed cleanly.
- Run: `30429706160` on `main` at commit `e01bc06e4eb4e91d4911d03d32d4821f38410bbd`.
## Root cause
The relink step re-resolves the dependency tree against the registry even though the artifact-restored `node_modules/` already satisfies everything — it only needs to be relinked locally. A transient upstream 5xx during that unnecessary network round-trip fails the whole job before the actual stress test (`integrationTests/stress/largeClone.test.mjs`) ever runs.
The same unguarded pattern (an artifact-restore step immediately followed by `npm install --ignore-scripts` to relink bins, plus a network-touching `npm rebuild re2`) also exists in `stress-tests.yaml` and both jobs of `integration-tests.yaml`.
## Fix (see linked PR)
- Relink steps switched to `npm rebuild --ignore-scripts --offline`, which recreates bin symlinks from the already-resolved local tree without touching the registry at all — verified locally that `npm rebuild --ignore-scripts --offline` restores a removed `node_modules/.bin` without any network access.
- Genuine registry-touching installs that can't go offline (`npm install` in the build jobs, `npm rebuild re2`'s native-binary download, the `harperdb@4` legacy install) get retry-with-backoff (4 attempts, 30s/60s/90s backoff) instead, so a transient 5xx/timeout no longer reds the job on the first hit.
- No `continue-on-error` / `|| true` was added anywhere — a genuinely broken dependency tree still fails loudly.
Contributor guide
Research direction
Start with .github/workflows/stress-large-data.yaml, then compare the corresponding artifact-restore and relink steps in stress-tests.yaml and integration-tests.yaml. Check integrationTests/stress/largeClone.test.mjs and the recorded run to verify that relinking is offline, registry-dependent steps retry, and genuine dependency failures still fail the workflows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100