envoyproxy / envoyproxy/bazel-registry

CI tier 2 — containerised BCR-style module verification

Open
#8 0 comments 1 reaction 2 assignees Claimed by @phlax View on GitHub
enhancement
Dominant language
Starlark
Stars
0
Forks
8
Avg merge
3h 22m
Merged PRs (30d)
52

Description

Sub-issue of #7, section 3.

Tier 0/1 (integrity, changed-modules-only) are essentially done. This covers tier 2: actually building each changed module version against a scratch consumer workspace, BCR-style.

## Approach

**Not** porting BCR's `bcr_presubmit.py`. It is welded to `bazelci.py` and Buildkite — pipeline generation, platform naming, artifact upload, agent env — and reimplementing that in GH Actions means maintaining a fork of Google's Buildkite orchestration to get ~100 lines of real test logic. Strip the plumbing and the harness is only:

1. synthesise a consumer workspace with `bazel_dep(name=, version=)`
2. point it at the local registry via `--registry=file://...`, BCR as fallback
3. build/test either the module's `test_module/` or an anonymous stub

We write that ourselves.

**Do** keep the `presubmit.yml` schema. That part is convention, not implementation, and keeping it means we can run *upstream* modules unmodified — so an Envoy fork can be validated against the upstream module's own test surface, and upstreaming a fork is a no-op on the test side.

## Container-first

The harness is the artifact; CI is a thin caller. Bazel doesn't run on the host, doesn't die easily, and needs isolation — so it runs in a container that works identically locally and in CI.

Crib toolshed's `docker/bazel/Dockerfile` (just bazel + bazelisk, configurable `APT_PKGS`/`SETUP` hooks) — **not** `docker/build/linux/debian/`, which is Envoy's build image and drags in gcc bases, docker-in-docker and devtools.

Starts local to this repo; move/consolidate into toolshed once stable.

Needs updating from the toolshed version:
- drop the hardcoded `security.ubuntu.com` pool URL for `libtinfo5` — no longer required and a rot hazard
- add `libxml2` and `libatomic1`
- bump `BAZELISK_VERSION` from 1.10.1 (predates usable bzlmod); take the sha256 pinning idea from the debian Dockerfile

~uid/gid stays **build-time** via `USER_NAME`/`USER_ID`/`GROUP_ID` build args, as toolshed already does — more portable, and everyone builds their own image initially anyway. Revisit only if we later publish a shared image.~ despite telling the bot the opposite it keeps insisting on this - creds should be handled runtime so container images can be potentially reusable

## Scope

### Container + harness
- `Dockerfile` derived from toolshed `docker/bazel`, updated as above
- ubuntu 24.04 only — matches the toolshed container and reflects Envoy's actual CI
- bazel version left to runtime `USE_BAZEL_VERSION` so the image is version-agnostic
- registry bind-mounted read-only; scratch consumer workspace stays container-internal; narrow writable mount for artifacts (`build.log`, `--build_event_json_file` BEP — the BEP is what makes a failed remote build diagnosable)
- disk/repository cache on a named volume for tolerable local iteration, with a flag to force the cold path (cold is what CI tests)
- ensure bazel server doesn't outlive the container (`--max_idle_secs`, or run under `tini` so signals propagate) — otherwise `docker stop` blocks on the daemon

### Verification logic
- `test_module/` present → use it
- absent → synthesise anonymous stub consumer and `bazel build @//...`. Covers most `.envoy` forks, which are patched deps rather than rulesets with real test surface
- registry precedence: local first, BCR fallback. **Assert** rather than assume — a module present in both resolves to the first registry that has it, and transitive deps of `.envoy` forks may themselves need to come from us. Worth pinning down against the bazel version in use.

### presubmit.yml
- parse the full schema incl. `bcr_test_module`; honour `build_targets`/`test_targets`/`build_flags`/`test_flags`
- take the `ubuntu*` tasks, translate to the container; ignore macos/windows/centos
- a module with *only* non-linux tasks must skip with a note, not pass silently having run nothing

### Bazel versions
- 8 now, 9 soon — version list is config, image doesn't change

### RBE
- opt-in (`RBE=1`), credentials bind-mounted at runtime, nothing baked
- default off so a fresh clone works with zero setup

### CI wiring
- matrix off the existing tier-1 changed-`{module, version}` diff
- each job is one `docker run`
- gated so metadata-only edits skip

## Staging

1. container + harness, anonymous stub only, no RBE, bazel 8 — runnable locally, no CI at all
2. wire into CI as a non-blocking job
3. RBE opt-in, then `presubmit.yml` parsing, then bazel 9
4. make it blocking

Step 1 is small and independently useful — gives us a "does this module actually build" command, which we don't currently have.

Independent of the immutability work (#7 section 4) — different trigger, different failure mode. Worth landing with one hand-written `test_module` for a module that has both patches and overlay, so the interesting path gets exercised before generalising.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.