mapbox / mapbox/node-pre-gyp

Binary downloads are not retried on transient HTTP failures

Open
#995 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
1.2k
Forks
271
Avg merge
23h 40m
Merged PRs (30d)
2

Description

## Summary

`node-pre-gyp install` makes exactly one attempt to download a pre-built binary. Any non-2xx response other than 403 fails immediately, so a transient server-side error (a 503 from S3, a 504 from a CDN, a 429 rate limit) forces a full source compile even though the same request would very likely succeed a second later.

There is no retry, backoff, or timeout anywhere in the download path.

## Observed behaviour

Installing `muhammara@6.0.5` (which downloads from GitHub releases) during a Docker build, with `node-pre-gyp@2.0.3`:

```
.../node_modules/muhammara install: [info] using node-pre-gyp@2.0.3
.../node_modules/muhammara install: [info] using node@24.19.0 | linux | x64
.../node_modules/muhammara install: [info] check checked for "/app/node_modules/.../muhammara.node" (not found)
.../node_modules/muhammara install: [log] GET https://github.com/julianhille/MuhammaraJS/releases/download/6.0.5/node-v137-linux-x64-glibc.tar.gz
.../node_modules/muhammara install: [error] install response status 504 Gateway Time-out on https://github.com/julianhille/MuhammaraJS/releases/download/6.0.5/node-v137-linux-x64-glibc.tar.gz
.../node_modules/muhammara install: [warn] Pre-built binaries not installable for muhammara@6.0.5 and node@24.19.0 (node-v137 ABI, glibc) (falling back to source compile with node-gyp)
.../node_modules/muhammara install: [warn] Hit error response status 504 Gateway Time-out on https://github.com/julianhille/MuhammaraJS/releases/download/6.0.5/node-v137-linux-x64-glibc.tar.gz
```

The 504 arrived roughly 11 seconds after the request was issued, and the install fell straight through to compiling from source.

## Impact

The whole point of a pre-built binary is to avoid a toolchain-dependent source build. A single transient gateway error defeats that:

- In CI and Docker builds, the fallback compile adds minutes to every affected install, or fails outright where no compiler or Python is present in the image.
- With `--fallback-to-build=false`, the install fails entirely rather than degrading.

## Second, related defect: the fallback diagnostic is dead code

`print_fallback_error` in `lib/install.js` branches on `err.statusCode` to choose between two messages. The status-aware branch produces the more useful output:

```js
if (err.statusCode !== undefined) {
full_message = 'Pre-built binaries not found for ' + ...
log.warn('Tried to download(' + err.statusCode + '): ' + opts.hosted_tarball);
...
} else {
full_message = 'Pre-built binaries not installable for ' + ...
log.warn('Hit error ' + err.message);
}
```

However, the error thrown for a non-2xx response is a bare `new Error(...)` with no `statusCode` property:

```js
throw new Error(`response status ${res.status} ${res.statusText} on ${sanitized}`);
```

So every HTTP failure takes the `else` branch, including a plain 404, which is the most common real-world case. The `Tried to download()` line never appears for an HTTP error, and users see "not installable" where "not found" was intended. This is visible in the trace above: a 504 reported as `Hit error` rather than `Tried to download(504)`.

Only the 403-without-credentials error and raw S3 SDK errors currently set `statusCode`.

## Expected behaviour

Transient failures should be retried a small, bounded number of times with backoff before falling back to a source build. Failures that cannot be fixed by retrying should continue to fall through immediately. That matters most for 404, which means there is genuinely no binary for this platform and ABI: it is the common path, and adding delay there would slow down every affected install.

## Environment

- node-pre-gyp 2.0.3
- Node.js 24.19.0, linux x64, glibc
- Binary host: GitHub releases
- Reproduces against any host returning a transient 5xx; not specific to `muhammara`

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the binary download path used by `node-pre-gyp install` and inspect `lib/install.js`, including `print_fallback_error` and the response-status error construction. Define completion as bounded retries with backoff for transient HTTP failures, immediate fallback for non-retryable responses such as 404, and status-aware diagnostics for HTTP errors; add or update the relevant download and fallback tests if present.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.