envoyproxy / envoyproxy/toolshed

ci: run Linux Bazel jobs in the bazel container

Open
#5,077 0 comments 1 reaction 2 assignees Claimed by @phlax View on GitHub
Dominant language
Python
Stars
12
Forks
24
Avg merge
6h 37m
Merged PRs (30d)
92

Description

Parked for now — to be picked up once other prerequisites are addressed.

## Goal

Run all **Linux** Bazel CI jobs inside the repo's Docker container (`docker/bazel/Dockerfile` via `docker/docker-compose.yml`) rather than directly on the `ubuntu-24.04` runner.

## Approach: build on the fly (not GHCR)

Decided against publishing the image to GHCR. Publishing on `main` means a PR that changes `docker/bazel/**` can't use an image built from its own change, so a PR-time build path is needed anyway — which leaves two mechanisms to maintain, plus registry permissions, tag lifecycle and retention. That's a second publishing pipeline standing next to the one that already exists.

Instead: build in-job, made cheap with buildx layer caching (`docker/setup-buildx-action` with `cache-from`/`cache-to` of `type=gha`). On a cache hit the build is seconds; the apt and bazelisk layers only rebuild when the Dockerfile or its build args change. Image size is small relative to the GHA cache budget.

`docker buildx bake` reads the compose file directly and integrates with buildx cache backends more cleanly than `docker compose build` — likely the better invocation.

## Current state

`.github/workflows/_bazel.yml` defines three jobs, all on `ubuntu-24.04` (or `ubuntu-24.04-arm`), each of which checks out, runs `bazelbuild/setup-bazelisk`, `apt-get install`s `debian-archive-keyring debootstrap qemu-user-static binfmt-support`, writes `repo.bazelrc` from `vars.TOOLSHED_CI_BAZELRC`, then runs bazel in `inputs.bazel-path`:

1. `bazel` — main job; also does bind-mounts (`/mnt/workspace`, `/mnt/cache`), RBE auth check, source tarball, artifact upload
2. `xcompile-x86-to-arm` — `ubuntu-24.04`
3. `xcompile-arm-to-x86` — `ubuntu-24.04-arm`

`.github/workflows/bazel.yml` callers in scope: `test`, `build`, `test-gcc`, `package-llvm` (all route through `_bazel.yml`).

## The container

`docker/docker-compose.yml` has a single `bazel` service, mounting `${SOURCE_DIR:-..}` at `/source` with `working_dir: /source/bazel`. Its entrypoint is built for **interactive local dev** — it drops into a `sudo -EHs` login shell. CI needs non-interactive execution, so the entrypoint/command needs overriding (e.g. `docker compose run --rm --entrypoint bash bazel -c '...'`), or a CI-specific compose override.

`docker/bazel/Dockerfile` is multi-stage: `base` → `bazel` (bazelisk to `/usr/local/bin/bazel`) → `full` (`$APT_PKGS`, `$SETUP`) → `extra` (`$EXTRA_APT_PKGS`, `$EXTRA_SETUP`).

## Requirements

1. All Linux Bazel invocations run in the container — the `bazel` job and both `xcompile-*` jobs.

2. The container provides the toolchain currently apt-installed on the runner (`debian-archive-keyring`, `debootstrap`, `qemu-user-static`, `binfmt-support`). Pass via the existing `APT_PKGS`/`EXTRA_APT_PKGS` build args rather than adding a `RUN apt-get` line — the Dockerfile is already parameterised for this. Remove the now-redundant "Install packages" workflow steps.

3. Bazel comes from the image, not `bazelbuild/setup-bazelisk`; drop that step for Linux jobs. The Dockerfile pins `BAZELISK_VERSION=1.10.1`, which is very old — `docker/build/linux/debian/Dockerfile` uses 1.26.0. Bump to match.

4. `repo.bazelrc` handling must still work. Written from `vars.TOOLSHED_CI_BAZELRC` to `${{ inputs.bazel-path }}/repo.bazelrc`. Writing on the host before invoking the container is fine given the bind mount, but verify paths line up: `working_dir` is `/source/bazel` and `inputs.bazel-path` is `bazel`, so host `bazel/repo.bazelrc` → container `/source/bazel/repo.bazelrc`.

5. Artifact paths must keep working. `_bazel.yml` uploads from `${{ inputs.bazel-path }}/bazel-bin/`; `bazel-bin` is a symlink into the output base. Ensure the host can follow it after the container exits, or adjust the upload step. **This is the most likely thing to break silently** — the upload succeeds with zero files rather than failing. Add a check that the artifact list is non-empty when `upload` is true.

6. UID/permissions. Compose takes `USER_NAME`/`USER_ID` build args and the Dockerfile `usermod`s `ubuntu` to match. GitHub runners are uid 1001 (`runner`), not the 1000 default, and the `bind-mounts` action chowns to `runner:runner`. Pass the actual runner uid/gid or every write to the bind-mounted workspace fails.

7. RBE must keep working. The `envoyproxy/toolshed/actions/github/container/auth` step runs on the host and gates `--config=rbe --config=bes`. Ensure the credentials/env it sets up are visible inside the container (`GITHUB_TOKEN` is already passed to the bazel step).

8. The bind-mounts step (`/mnt/workspace` → workspace, `/mnt/cache` → `/home/runner/.cache`, for disk space) — decide deliberately whether it is still needed once containerised. Keep or remove; don't leave it half-wired.

## Main risk: binfmt / qemu

The cross-compile jobs depend on `qemu-user-static`/`binfmt-support`. binfmt_misc registration is **kernel-level and host-global** — installing qemu inside the container does not register handlers. If the container can't see the host's binfmt handlers, the cross-compile tests fail. Investigate and address explicitly; may require keeping the host-level qemu install, or registering binfmt on the host before starting the container.

## Out of scope

- `test-macos` (macos-14) — leave entirely alone
- Any `--repository_cache` / Bazel caching work — separate follow-up
- The `request` and `status` coordination jobs
- `package-macos` — runs on Linux and apt-installs `llvm-18`, but does not use `_bazel.yml`. Separate in-flight work (branch `ci-macos`) moves this packaging into Bazel proper (`//compile:libcxx_darwin_aarch64`, `//sysroot:sysroot_macos_arm64`), which would make the job redundant. Do not restructure it here; just state in the PR whether it was containerised or left, and why.

## Verification

- All Linux Bazel jobs pass
- Artifacts still produced and uploaded non-empty (`bazel-artifacts`, `bazel-llvm-*-artifacts`)
- Cross-compile jobs still work (see binfmt risk above)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.