envoyproxy / envoyproxy/toolshed
Fix bzlmod cross-compile (wrong-arch libc++/sysroot, slow LLVM download) without breaking WORKSPACE
- Dominant language
- Python
- Stars
- 12
- Forks
- 24
- Avg merge
- 6h 37m
- Merged PRs (30d)
- 92
Description
## Goal / acceptance criteria
On branch `llvm-min` (PR #4824, head fork `phlax/toolshed`), fix the **bzlmod** cross-compile so it works, without breaking WORKSPACE mode.
Acceptance:
1. **WORKSPACE mode is not broken** — native builds AND cross-compile continue to work (WORKSPACE x-compile currently works and is fast; it must stay working).
2. **bzlmod cross-compile works** — the cross-compile test targets build and pass under bzlmod.
Validate (run in both failing and fixed states):
```
cd bazel
# bzlmod (default)
bazel test --platforms=@toolchains_llvm//platforms:linux-aarch64 \
//compile/test:cross_compile_aarch64_test \
//compile/test:cross_compile_aarch64_unwind_test
bazel test --platforms=@toolchains_llvm//platforms:linux-x86_64 \
//compile/test:cross_compile_x86_64_test \
//compile/test:cross_compile_x86_64_unwind_test
# WORKSPACE mode - must remain green
```
## Current state
Recent commits on this branch reworked LLVM toolchain consumption to use a minimal prebuilt LLVM artifact and a host-arch `@llvm_toolchain_llvm` repo (no `alias()`), and wired cross-compile libc++ via a patched `toolchains_llvm` (`cxx_cross_lib` + `sysroot` per target pair).
- **WORKSPACE mode:** works and is FAST — notably does NOT download the old full upstream LLVM prebuild. Must not regress.
- **bzlmod mode:** very SLOW (appears to still download/resolve a large LLVM distribution) and FAILS at the end — appears to resolve the WRONG-ARCH cross libc++ libs and/or sysroot for the target (symptoms consistent with picking host-arch instead of target-arch, or the x-compile libc++ `-isystem`/`-L` paths pointing at the wrong tree).
## Strong lead to investigate first
`toolchain_roots`: the repo patches in the cross-compile libc++ wiring (the `cxx_cross_lib` block in `toolchain/internal/configure.bzl` that adds `-nostdinc++ -isystem /include/c++/v1 -isystem /include`, `-L/lib -l:libc++.a -l:libc++abi.a`, and auto-selects libc++ stdlib for target pairs). Strong suspicion: when `toolchain_roots` is set (to point the toolchain at the minimal repo), the x-compile patch's cross-libc++ wiring does NOT get applied / gets bypassed in the bzlmod path — so the WORKSPACE path works while bzlmod does not.
Trace concretely:
1. Diff how the toolchain is configured in WORKSPACE mode (`bazel/toolchains_llvm.bzl` `setup_llvm_toolchain`) vs bzlmod mode (`bazel/MODULE.bazel` `llvm.toolchain` / `llvm.cxx_cross_lib` / `llvm.sysroot` tags, plus the vendored `toolchains_llvm` module + `x_compile.patch`). Identify every difference in: `toolchain_roots`, `cxx_cross_lib`, `sysroot`, `llvm_version`/`extra_llvm_distributions`, and whether the minimal repo is used.
2. Confirm whether the bzlmod extension path threads `cxx_cross_lib` and `sysroot` (and the minimal `toolchain_roots`) through to `llvm_config_impl` the same way WORKSPACE does. Check the extension attr scrubbing (`_attrs.pop("toolchain_roots"...)`, `_attrs.pop("cxx_cross_lib"...)`, `_attrs.pop("sysroot"...)`) and per-target resolution in `configure.bzl`.
3. Determine why bzlmod is slow / still downloading a large LLVM distribution while WORKSPACE is not — i.e. whether bzlmod fetches the full upstream LLVM prebuild instead of the minimal artifact, and whether that wrong distribution causes the wrong-arch libc++/sysroot resolution.
4. Capture, with `-s`/`--sandbox_debug` evidence, which `-isystem`, `-L`, `--sysroot`, `--target` flags bzlmod emits for the cross target, and how they differ from the working WORKSPACE invocation.
## Hard constraints
- `@llvm_toolchain_llvm` MUST resolve to the host-arch minimal repo and MUST NOT use Bazel `alias()`.
- Do NOT regress WORKSPACE mode (native or cross).
- Do NOT regress native (non-cross) builds in either mode.
- Keep the fix minimal; explain root cause with concrete flag/path evidence in the PR description.
## Likely files
`bazel/MODULE.bazel`, `bazel/toolchains_llvm.bzl`, `bazel/compile/llvm_minimal.bzl`, `bazel/versions.bzl`, and the vendored `bazel-registry/modules/toolchains_llvm/*/patches/x_compile.patch` (+ the module's `configure.bzl`/extension). Focus on the bzlmod-vs-WORKSPACE divergence around `toolchain_roots` and the cross-libc++ wiring.
## Context
- Prior coding-agent session on this stalled (fixed things locally, pushed nothing, then lost the thread). Filing as an issue instead.
- Related PR: #4824. Branch: `phlax:llvm-min`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the listed bzlmod and WORKSPACE tests with -s/--sandbox_debug, then compare bazel/MODULE.bazel and bazel/toolchains_llvm.bzl. Trace toolchain_roots, cxx_cross_lib, and sysroot through bazel-registry/modules/toolchains_llvm/.../configure.bzl and x_compile.patch; inspect compile/llvm_minimal.bzl and versions.bzl for distribution selection. Done means native and cross tests pass in both modes, with the minimal host-arch LLVM and no regressions.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100