envoyproxy / envoyproxy/toolshed
ci: run Linux Bazel jobs in the bazel container
- Dominant language
- Python
- Stars
- 12
- Forks
- 24
- Avg merge
- 6h 37m
- Merged PRs (30d)
- 92
Description
Parked for now — to be picked up once other prerequisites are addressed.
## Goal
Run all **Linux** Bazel CI jobs inside the repo's Docker container (`docker/bazel/Dockerfile` via `docker/docker-compose.yml`) rather than directly on the `ubuntu-24.04` runner.
## Approach: build on the fly (not GHCR)
Decided against publishing the image to GHCR. Publishing on `main` means a PR that changes `docker/bazel/**` can't use an image built from its own change, so a PR-time build path is needed anyway — which leaves two mechanisms to maintain, plus registry permissions, tag lifecycle and retention. That's a second publishing pipeline standing next to the one that already exists.
Instead: build in-job, made cheap with buildx layer caching (`docker/setup-buildx-action` with `cache-from`/`cache-to` of `type=gha`). On a cache hit the build is seconds; the apt and bazelisk layers only rebuild when the Dockerfile or its build args change. Image size is small relative to the GHA cache budget.
`docker buildx bake` reads the compose file directly and integrates with buildx cache backends more cleanly than `docker compose build` — likely the better invocation.
## Current state
`.github/workflows/_bazel.yml` defines three jobs, all on `ubuntu-24.04` (or `ubuntu-24.04-arm`), each of which checks out, runs `bazelbuild/setup-bazelisk`, `apt-get install`s `debian-archive-keyring debootstrap qemu-user-static binfmt-support`, writes `repo.bazelrc` from `vars.TOOLSHED_CI_BAZELRC`, then runs bazel in `inputs.bazel-path`:
1. `bazel` — main job; also does bind-mounts (`/mnt/workspace`, `/mnt/cache`), RBE auth check, source tarball, artifact upload
2. `xcompile-x86-to-arm` — `ubuntu-24.04`
3. `xcompile-arm-to-x86` — `ubuntu-24.04-arm`
`.github/workflows/bazel.yml` callers in scope: `test`, `build`, `test-gcc`, `package-llvm` (all route through `_bazel.yml`).
## The container
`docker/docker-compose.yml` has a single `bazel` service, mounting `${SOURCE_DIR:-..}` at `/source` with `working_dir: /source/bazel`. Its entrypoint is built for **interactive local dev** — it drops into a `sudo -EHs` login shell. CI needs non-interactive execution, so the entrypoint/command needs overriding (e.g. `docker compose run --rm --entrypoint bash bazel -c '...'`), or a CI-specific compose override.
`docker/bazel/Dockerfile` is multi-stage: `base` → `bazel` (bazelisk to `/usr/local/bin/bazel`) → `full` (`$APT_PKGS`, `$SETUP`) → `extra` (`$EXTRA_APT_PKGS`, `$EXTRA_SETUP`).
## Requirements
1. All Linux Bazel invocations run in the container — the `bazel` job and both `xcompile-*` jobs.
2. The container provides the toolchain currently apt-installed on the runner (`debian-archive-keyring`, `debootstrap`, `qemu-user-static`, `binfmt-support`). Pass via the existing `APT_PKGS`/`EXTRA_APT_PKGS` build args rather than adding a `RUN apt-get` line — the Dockerfile is already parameterised for this. Remove the now-redundant "Install packages" workflow steps.
3. Bazel comes from the image, not `bazelbuild/setup-bazelisk`; drop that step for Linux jobs. The Dockerfile pins `BAZELISK_VERSION=1.10.1`, which is very old — `docker/build/linux/debian/Dockerfile` uses 1.26.0. Bump to match.
4. `repo.bazelrc` handling must still work. Written from `vars.TOOLSHED_CI_BAZELRC` to `${{ inputs.bazel-path }}/repo.bazelrc`. Writing on the host before invoking the container is fine given the bind mount, but verify paths line up: `working_dir` is `/source/bazel` and `inputs.bazel-path` is `bazel`, so host `bazel/repo.bazelrc` → container `/source/bazel/repo.bazelrc`.
5. Artifact paths must keep working. `_bazel.yml` uploads from `${{ inputs.bazel-path }}/bazel-bin/`; `bazel-bin` is a symlink into the output base. Ensure the host can follow it after the container exits, or adjust the upload step. **This is the most likely thing to break silently** — the upload succeeds with zero files rather than failing. Add a check that the artifact list is non-empty when `upload` is true.
6. UID/permissions. Compose takes `USER_NAME`/`USER_ID` build args and the Dockerfile `usermod`s `ubuntu` to match. GitHub runners are uid 1001 (`runner`), not the 1000 default, and the `bind-mounts` action chowns to `runner:runner`. Pass the actual runner uid/gid or every write to the bind-mounted workspace fails.
7. RBE must keep working. The `envoyproxy/toolshed/actions/github/container/auth` step runs on the host and gates `--config=rbe --config=bes`. Ensure the credentials/env it sets up are visible inside the container (`GITHUB_TOKEN` is already passed to the bazel step).
8. The bind-mounts step (`/mnt/workspace` → workspace, `/mnt/cache` → `/home/runner/.cache`, for disk space) — decide deliberately whether it is still needed once containerised. Keep or remove; don't leave it half-wired.
## Main risk: binfmt / qemu
The cross-compile jobs depend on `qemu-user-static`/`binfmt-support`. binfmt_misc registration is **kernel-level and host-global** — installing qemu inside the container does not register handlers. If the container can't see the host's binfmt handlers, the cross-compile tests fail. Investigate and address explicitly; may require keeping the host-level qemu install, or registering binfmt on the host before starting the container.
## Out of scope
- `test-macos` (macos-14) — leave entirely alone
- Any `--repository_cache` / Bazel caching work — separate follow-up
- The `request` and `status` coordination jobs
- `package-macos` — runs on Linux and apt-installs `llvm-18`, but does not use `_bazel.yml`. Separate in-flight work (branch `ci-macos`) moves this packaging into Bazel proper (`//compile:libcxx_darwin_aarch64`, `//sysroot:sysroot_macos_arm64`), which would make the job redundant. Do not restructure it here; just state in the PR whether it was containerised or left, and why.
## Verification
- All Linux Bazel jobs pass
- Artifacts still produced and uploaded non-empty (`bazel-artifacts`, `bazel-llvm-*-artifacts`)
- Cross-compile jobs still work (see binfmt risk above)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.