envoyproxy / envoyproxy/toolshed

Fix bzlmod cross-compile (wrong-arch libc++/sysroot, slow LLVM download) without breaking WORKSPACE

Open
#4,829 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
24
Avg merge
6h 37m
Merged PRs (30d)
92

Description

## Goal / acceptance criteria

On branch `llvm-min` (PR #4824, head fork `phlax/toolshed`), fix the **bzlmod** cross-compile so it works, without breaking WORKSPACE mode.

Acceptance:
1. **WORKSPACE mode is not broken** — native builds AND cross-compile continue to work (WORKSPACE x-compile currently works and is fast; it must stay working).
2. **bzlmod cross-compile works** — the cross-compile test targets build and pass under bzlmod.

Validate (run in both failing and fixed states):

```
cd bazel
# bzlmod (default)
bazel test --platforms=@toolchains_llvm//platforms:linux-aarch64 \
//compile/test:cross_compile_aarch64_test \
//compile/test:cross_compile_aarch64_unwind_test
bazel test --platforms=@toolchains_llvm//platforms:linux-x86_64 \
//compile/test:cross_compile_x86_64_test \
//compile/test:cross_compile_x86_64_unwind_test
# WORKSPACE mode - must remain green
```

## Current state

Recent commits on this branch reworked LLVM toolchain consumption to use a minimal prebuilt LLVM artifact and a host-arch `@llvm_toolchain_llvm` repo (no `alias()`), and wired cross-compile libc++ via a patched `toolchains_llvm` (`cxx_cross_lib` + `sysroot` per target pair).

- **WORKSPACE mode:** works and is FAST — notably does NOT download the old full upstream LLVM prebuild. Must not regress.
- **bzlmod mode:** very SLOW (appears to still download/resolve a large LLVM distribution) and FAILS at the end — appears to resolve the WRONG-ARCH cross libc++ libs and/or sysroot for the target (symptoms consistent with picking host-arch instead of target-arch, or the x-compile libc++ `-isystem`/`-L` paths pointing at the wrong tree).

## Strong lead to investigate first

`toolchain_roots`: the repo patches in the cross-compile libc++ wiring (the `cxx_cross_lib` block in `toolchain/internal/configure.bzl` that adds `-nostdinc++ -isystem /include/c++/v1 -isystem /include`, `-L/lib -l:libc++.a -l:libc++abi.a`, and auto-selects libc++ stdlib for target pairs). Strong suspicion: when `toolchain_roots` is set (to point the toolchain at the minimal repo), the x-compile patch's cross-libc++ wiring does NOT get applied / gets bypassed in the bzlmod path — so the WORKSPACE path works while bzlmod does not.

Trace concretely:
1. Diff how the toolchain is configured in WORKSPACE mode (`bazel/toolchains_llvm.bzl` `setup_llvm_toolchain`) vs bzlmod mode (`bazel/MODULE.bazel` `llvm.toolchain` / `llvm.cxx_cross_lib` / `llvm.sysroot` tags, plus the vendored `toolchains_llvm` module + `x_compile.patch`). Identify every difference in: `toolchain_roots`, `cxx_cross_lib`, `sysroot`, `llvm_version`/`extra_llvm_distributions`, and whether the minimal repo is used.
2. Confirm whether the bzlmod extension path threads `cxx_cross_lib` and `sysroot` (and the minimal `toolchain_roots`) through to `llvm_config_impl` the same way WORKSPACE does. Check the extension attr scrubbing (`_attrs.pop("toolchain_roots"...)`, `_attrs.pop("cxx_cross_lib"...)`, `_attrs.pop("sysroot"...)`) and per-target resolution in `configure.bzl`.
3. Determine why bzlmod is slow / still downloading a large LLVM distribution while WORKSPACE is not — i.e. whether bzlmod fetches the full upstream LLVM prebuild instead of the minimal artifact, and whether that wrong distribution causes the wrong-arch libc++/sysroot resolution.
4. Capture, with `-s`/`--sandbox_debug` evidence, which `-isystem`, `-L`, `--sysroot`, `--target` flags bzlmod emits for the cross target, and how they differ from the working WORKSPACE invocation.

## Hard constraints

- `@llvm_toolchain_llvm` MUST resolve to the host-arch minimal repo and MUST NOT use Bazel `alias()`.
- Do NOT regress WORKSPACE mode (native or cross).
- Do NOT regress native (non-cross) builds in either mode.
- Keep the fix minimal; explain root cause with concrete flag/path evidence in the PR description.

## Likely files

`bazel/MODULE.bazel`, `bazel/toolchains_llvm.bzl`, `bazel/compile/llvm_minimal.bzl`, `bazel/versions.bzl`, and the vendored `bazel-registry/modules/toolchains_llvm/*/patches/x_compile.patch` (+ the module's `configure.bzl`/extension). Focus on the bzlmod-vs-WORKSPACE divergence around `toolchain_roots` and the cross-libc++ wiring.

## Context

- Prior coding-agent session on this stalled (fixed things locally, pushed nothing, then lost the thread). Filing as an issue instead.
- Related PR: #4824. Branch: `phlax:llvm-min`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the listed bzlmod and WORKSPACE tests with -s/--sandbox_debug, then compare bazel/MODULE.bazel and bazel/toolchains_llvm.bzl. Trace toolchain_roots, cxx_cross_lib, and sysroot through bazel-registry/modules/toolchains_llvm/.../configure.bzl and x_compile.patch; inspect compile/llvm_minimal.bzl and versions.bzl for distribution selection. Done means native and cross tests pass in both modes, with the minimal host-arch LLVM and no regressions.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.