bazel-contrib / bazel-contrib/bazel-lib

Add mirror fallbacks for bazel_lib toolchain binary downloads

Open
#1,295 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
182
Forks
134
Avg merge
1d 46m
Merged PRs (30d)
1

Description

## Summary

`bazel_lib`'s toolchain repository rules download prebuilt binaries from
GitHub releases via a single hardcoded URL, with no `mirror.bazel.build` /
`urls`-list fallback. A GitHub outage (or unauthenticated-request rejection)
breaks any cold `--repository_cache` build that uses these toolchains.

## Evidence

`lib/private/coreutils_toolchain.bzl` (same pattern in 3.2.2, 3.7.1 and main):

```python
# line ~263
url = "https://github.com/uutils/coreutils/releases/download/{}/{}".format(
rctx.attr.version, filename)
rctx.download_and_extract(
url = url, # single URL, no fallback
integrity = COREUTILS_VERSIONS[rctx.attr.version][platform]["sha256"],
)
```

- Single `url`, not a `urls` list; no `mirror.bazel.build` entry.
- `integrity` is present, so a mirror would be trustless (Bazel verifies bytes).
- `rctx.download_and_extract` goes through Bazel's downloader, so
`--downloader_config` `rewrite`/`block` rules apply.

## Affected downloads

- `coreutils` (uutils/coreutils) — in `bazel_lib`.
- `tar`, `jq`, `yq` — split into their own modules (`tar.bzl`, `jq.bzl`,
`yq.bzl`) in 3.0; same single-URL github download pattern.
- Consumers: e.g. `rules_js` uses coreutils for `chmod`/`mkdir`/`cp` on the
npm fetch path (`npm/private/npm_import.bzl`, `utils.bzl`,
`npm_translate_lock_state.bzl`), so every npm extract needs it.

## Contrast: `rules_java`

`remotejdk` downloads include `mirror.bazel.build` URLs; these `bazel_lib`
toolchain downloads do not.

## Reproduction

```bash
cat > /tmp/block-github.cfg <<'EOF'
block github.com
block codeload.github.com
block objects.githubusercontent.com
block release-assets.githubusercontent.com
block raw.githubusercontent.com
EOF

# A target that pulls a bazel_lib toolchain, cold cache, github blocked.
# For example, in Gerrit (a rules_js consumer):
rm -rf /tmp/ob /tmp/repo
bazelisk --output_base=/tmp/ob fetch --repository_cache=/tmp/repo \
--downloader_config=/tmp/block-github.cfg \
//polygerrit-ui/app:node_modules/resemblejs/dir
# -> Error downloading [https://github.com/uutils/coreutils/releases/download/...]:
# Configured URL rewriter blocked all URLs
```

## Request

Give the toolchain downloads (coreutils, tar, jq, yq) a fallback URL:
accept a `urls` list / expose a mirror-URL attribute, and optionally add a
`mirror.bazel.build` URL if Bazel infra can host the artifacts. `integrity`
is already set, so a fallback is verified and safe.

## Environment

- `bazel_lib` 3.2.2 / `aspect_bazel_lib` 2.22.5 (same single-URL pattern in
3.7.1 and main)
- Bazel 9.1.0
- consumer: `aspect_rules_js`

## References

- Bazel #21687 — [Mirror] bazel-lib copy_to_directory-linux_amd64:
https://github.com/bazelbuild/bazel/issues/21687
- bazel-lib #51 — Mirror releases to bcr (closed; scoped to bazel-lib's own
releases, not third-party coreutils):
https://github.com/bazel-contrib/bazel-lib/issues/51
- bazel-lib #671 — register_coreutils_toolchains version attribute:
https://github.com/bazel-contrib/bazel-lib/issues/671
- Source: https://github.com/bazel-contrib/bazel-lib/blob/main/lib/private/coreutils_toolchain.bzl
- Bazel downloader config (rewrite/block):
https://bazel.build/reference/command-line-reference#flag--downloader_config

Contributor guide

Open the contributing guide

Research direction

Start with lib/private/coreutils_toolchain.bzl and compare the download patterns in tar.bzl, jq.bzl, and yq.bzl. Run the blocked-GitHub reproduction with a cold repository cache to confirm the failure. Done means the toolchain downloads have a verified fallback URL path and the affected build succeeds when GitHub URLs are blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
github
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.