ci-node: a transient registry timeout fails the whole run; the install step has no retry
- Dominant language
- Shell
- Stars
- 1
- Forks
- 2
- Avg merge
- 12h 53m
- Merged PRs (30d)
- 23
Description
## What happened
A single npm registry timeout during dependency install turned a green PR red in a
consumer repo. `ci` is the required status check in every repo using
`protection/`, so the flake blocked the merge until someone manually re-ran it.
Observed in `celo-org/mondeto`, [run 32274621217](https://github.com/celo-org/mondeto/actions/runs/32274621217):
```
Progress: resolved 1329, reused 0, downloaded 618, added 625
ETIMEDOUT request to https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz failed, reason:
FetchError: request to https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz failed
at ClientRequest. (/home/runner/setup-pnpm/node_modules/.pnpm/pnpm@8.10.0/...)
##[error]Process completed with exit code 1.
```
Failing step: **`Install (frozen lockfile)`**. Everything downstream (lint, typecheck,
test) never ran.
**Proven transient with a control run:** I re-ran the identical commit with no code
change. Green end to end — `Install: success, Lint: success, Typecheck: success,
Test: success`. Nothing about the diff caused it.
## Root cause
`.github/workflows/ci-node.yml:84-91` runs the install with no retry:
```yaml
- name: Install (frozen lockfile)
run: |
case "${{ steps.detect.outputs.pm }}" in
pnpm) pnpm install --frozen-lockfile ;;
...
```
**Worth being precise, because it changes the fix:** this is *not* a case of "no retries
at all". pnpm already retries at the fetch layer — verified against the docs rather than
assumed, `fetchRetries` defaults to **2** with exponential backoff (`fetchRetryFactor` 10,
min 10s, max 60s, `fetchTimeout` 60s). So this tarball was already attempted three times
and still timed out. Fetch-level retry exists and was not enough.
What is missing is a **step-level** retry: once install exits 1, the run is dead and only
a human can restart it.
## Impact
Every repo calling this workflow, on every PR. Two things make it more than an
annoyance:
1. `ci` is a **required status check** under `protection/`, so a network blip is
indistinguishable from a real failure at the merge button.
2. The rulesets also require **branch up to date with main**. In a busy queue, a
spurious red costs a re-run *plus* a rebase and a second full run.
**Scope of this evidence:** one confirmed occurrence, reproduced-as-transient by control.
I did not measure a rate — job-level data for older runs is no longer retained, so I
can't say whether this is monthly or weekly. The argument for fixing it is structural
(every repo × every PR × a single point of failure with no recovery), not frequency.
## Suggested fix
Two options, and I'd lean to the second:
1. **Wrap the step in a retry action** (`nick-fields/retry@v3`, 2-3 attempts). Covers all
four package managers uniformly, including `npm ci` and `yarn`, which have their own
retry semantics.
2. **Raise the fetch budget via env**, e.g. `NPM_CONFIG_FETCH_RETRIES: 5` on the step.
No new action dependency and it is read by npm, pnpm and yarn alike. Smaller change,
though it only helps registry fetches — it does nothing for a failure later in install.
**Trade-off to decide either way:** a retry makes a genuinely broken lockfile fail
*slower* (3× the install time before the real error surfaces). That is probably an
acceptable price, since `ERR_PNPM_OUTDATED_LOCKFILE` is a fast failure, but it is a
deliberate choice rather than a free win. If option 1 is taken, restricting retries to
network-class errors would avoid it entirely.
Retrying `--frozen-lockfile` is safe on correctness grounds: the install is deterministic
and idempotent, so a retry cannot resolve different versions or mask a lockfile mismatch.
## Acceptance
- [ ] A transient registry failure during install no longer fails the run on the first
attempt
- [ ] The retry is bounded and its count is visible in the log, so a genuine install
failure is still obviously a failure rather than a hang
- [ ] Decision recorded on whether retries are network-scoped or blanket
Refs celo-org/mondeto#248, where this surfaced.
Contributor guide
Research direction
Start with .github/workflows/ci-node.yml:84-91 and inspect the Install (frozen lockfile) step, including its package-manager case branches. Review the two proposed retry scopes and their bounded-attempt trade-off. Done means a transient registry failure does not fail on the first attempt, retry count is visible and bounded, and the network-scoped versus blanket decision is recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, shell
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100