HarperFast / HarperFast/harper-pro

Nightly Large-Data Stress workflow reds on transient npm-registry 5xx during bin relink

Open
#630 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary
The `Large-Data Stress Tests` nightly (`.github/workflows/stress-large-data.yaml`) failed on 2026-07-29 without running a single test. The failure was in the `Relink bin scripts` step of the `large-clone` job, which runs `npm install --ignore-scripts` against the artifact-restored `node_modules/` purely to recreate `node_modules/.bin` symlinks after the build artifact is downloaded.

```
npm error code E503
npm error 503 Service Unavailable - GET https://registry.npmjs.org/@babel%2fparser - Service Unavailable
##[error]Process completed with exit code 1.
```

## Evidence it was a registry blip, not a runner/network problem
- The sibling matrix job in the same run, `large-catchup` (same self-hosted runner pool, same build artifact, same commit), completed successfully in 29m3s.
- The two prior nightlies (2026-07-27, 2026-07-28) passed cleanly.
- Run: `30429706160` on `main` at commit `e01bc06e4eb4e91d4911d03d32d4821f38410bbd`.

## Root cause
The relink step re-resolves the dependency tree against the registry even though the artifact-restored `node_modules/` already satisfies everything — it only needs to be relinked locally. A transient upstream 5xx during that unnecessary network round-trip fails the whole job before the actual stress test (`integrationTests/stress/largeClone.test.mjs`) ever runs.

The same unguarded pattern (an artifact-restore step immediately followed by `npm install --ignore-scripts` to relink bins, plus a network-touching `npm rebuild re2`) also exists in `stress-tests.yaml` and both jobs of `integration-tests.yaml`.

## Fix (see linked PR)
- Relink steps switched to `npm rebuild --ignore-scripts --offline`, which recreates bin symlinks from the already-resolved local tree without touching the registry at all — verified locally that `npm rebuild --ignore-scripts --offline` restores a removed `node_modules/.bin` without any network access.
- Genuine registry-touching installs that can't go offline (`npm install` in the build jobs, `npm rebuild re2`'s native-binary download, the `harperdb@4` legacy install) get retry-with-backoff (4 attempts, 30s/60s/90s backoff) instead, so a transient 5xx/timeout no longer reds the job on the first hit.
- No `continue-on-error` / `|| true` was added anywhere — a genuinely broken dependency tree still fails loudly.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/stress-large-data.yaml, then compare the corresponding artifact-restore and relink steps in stress-tests.yaml and integration-tests.yaml. Check integrationTests/stress/largeClone.test.mjs and the recorded run to verify that relinking is offline, registry-dependent steps retry, and genuine dependency failures still fail the workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.