celo-org / celo-org/pm-kit

ci-node: a transient registry timeout fails the whole run; the install step has no retry

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
1
Forks
2
Avg merge
12h 53m
Merged PRs (30d)
23

Description

## What happened

A single npm registry timeout during dependency install turned a green PR red in a
consumer repo. `ci` is the required status check in every repo using
`protection/`, so the flake blocked the merge until someone manually re-ran it.

Observed in `celo-org/mondeto`, [run 32274621217](https://github.com/celo-org/mondeto/actions/runs/32274621217):

```
Progress: resolved 1329, reused 0, downloaded 618, added 625
ETIMEDOUT request to https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz failed, reason:

FetchError: request to https://registry.npmjs.org/@types/node/-/node-20.19.43.tgz failed
at ClientRequest. (/home/runner/setup-pnpm/node_modules/.pnpm/pnpm@8.10.0/...)
##[error]Process completed with exit code 1.
```

Failing step: **`Install (frozen lockfile)`**. Everything downstream (lint, typecheck,
test) never ran.

**Proven transient with a control run:** I re-ran the identical commit with no code
change. Green end to end — `Install: success, Lint: success, Typecheck: success,
Test: success`. Nothing about the diff caused it.

## Root cause

`.github/workflows/ci-node.yml:84-91` runs the install with no retry:

```yaml
- name: Install (frozen lockfile)
run: |
case "${{ steps.detect.outputs.pm }}" in
pnpm) pnpm install --frozen-lockfile ;;
...
```

**Worth being precise, because it changes the fix:** this is *not* a case of "no retries
at all". pnpm already retries at the fetch layer — verified against the docs rather than
assumed, `fetchRetries` defaults to **2** with exponential backoff (`fetchRetryFactor` 10,
min 10s, max 60s, `fetchTimeout` 60s). So this tarball was already attempted three times
and still timed out. Fetch-level retry exists and was not enough.

What is missing is a **step-level** retry: once install exits 1, the run is dead and only
a human can restart it.

## Impact

Every repo calling this workflow, on every PR. Two things make it more than an
annoyance:

1. `ci` is a **required status check** under `protection/`, so a network blip is
indistinguishable from a real failure at the merge button.
2. The rulesets also require **branch up to date with main**. In a busy queue, a
spurious red costs a re-run *plus* a rebase and a second full run.

**Scope of this evidence:** one confirmed occurrence, reproduced-as-transient by control.
I did not measure a rate — job-level data for older runs is no longer retained, so I
can't say whether this is monthly or weekly. The argument for fixing it is structural
(every repo × every PR × a single point of failure with no recovery), not frequency.

## Suggested fix

Two options, and I'd lean to the second:

1. **Wrap the step in a retry action** (`nick-fields/retry@v3`, 2-3 attempts). Covers all
four package managers uniformly, including `npm ci` and `yarn`, which have their own
retry semantics.
2. **Raise the fetch budget via env**, e.g. `NPM_CONFIG_FETCH_RETRIES: 5` on the step.
No new action dependency and it is read by npm, pnpm and yarn alike. Smaller change,
though it only helps registry fetches — it does nothing for a failure later in install.

**Trade-off to decide either way:** a retry makes a genuinely broken lockfile fail
*slower* (3× the install time before the real error surfaces). That is probably an
acceptable price, since `ERR_PNPM_OUTDATED_LOCKFILE` is a fast failure, but it is a
deliberate choice rather than a free win. If option 1 is taken, restricting retries to
network-class errors would avoid it entirely.

Retrying `--frozen-lockfile` is safe on correctness grounds: the install is deterministic
and idempotent, so a retry cannot resolve different versions or mask a lockfile mismatch.

## Acceptance

- [ ] A transient registry failure during install no longer fails the run on the first
attempt
- [ ] The retry is bounded and its count is visible in the log, so a genuine install
failure is still obviously a failure rather than a hang
- [ ] Decision recorded on whether retries are network-scoped or blanket

Refs celo-org/mondeto#248, where this surfaced.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/ci-node.yml:84-91 and inspect the Install (frozen lockfile) step, including its package-manager case branches. Review the two proposed retry scopes and their bounded-attempt trade-off. Done means a transient registry failure does not fail on the first attempt, retry count is visible and bounded, and the network-scoped versus blanket decision is recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell
Domain
ci-cd
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.