NVIDIA / NVIDIA/OpenShell

feat(ci): add a local Linux validation task so non-Linux contributors can exercise Linux-gated Rust code

Open
#3,039 3 comments 0 reactions 1 assignee View on GitHub

@SDAChess is already working on this.

Since Sep 18, 2026.

area:build Improvement
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

User Story

As a contributor developing OpenShell on macOS or Windows, I want a mise task that compiles, lints, and tests the workspace for a Linux target locally, so that I catch failures in #[cfg(target_os = "linux")] code before pushing — instead of discovering them only in the branch checks or the merge queue.

Problem Statement

mise run ci / mise run test run against the host target. The workspace contains 434 target_os = "linux" gated lines across 42 files and 12 crates (heaviest by file count: openshell-sandbox, openshell-supervisor-network, openshell-driver-vm, openshell-supervisor-process, openshell-driver-podman). On a macOS or Windows host, cargo's clippy/check/test never see those blocks. Windows contributors do run Rust checks locally (the rust:lint / test:rust tasks have run_windows -> windows-msvc.ps1 for the native MSVC target), but that target is windows, so the Linux-gated code is still never compiled. There is no host-runnable path that compiles the Linux-gated code on a non-Linux machine.

CI does exercise it, but through a path contributors cannot run locally. In .github/workflows/branch-checks.yml the Rust lane runs inside a Nix devShell on a matrix of x86_64-linux | aarch64-linux | aarch64-darwin:

shell: nix develop .#devShells.${{ matrix.system }}.default -c bash ...
# Format: cargo fmt --all -- --check
# Lint:   cargo clippy --workspace --all-targets -- -D warnings
# Test:   cargo nextest run --profile ci --workspace --features openshell-server/test-support

The aarch64-darwin lane mirrors a non-Linux host and passes; the Linux-gated code compiles only on the two Linux runners. Locally, nix develop .#devShells.x86_64-linux... cannot realize a Linux devShell on a non-Linux host without a Linux builder. A Linux CI image exists (deploy/docker/Dockerfile.ci -> ghcr.io/nvidia/openshell/ci, built by ci-image.yml, used by the go/python/license/markdown/cargo-deny branch-checks jobs), but no task runs the Rust checks — or any checks — inside a Linux container. This is a concrete slice of the local<->CI parity gap tracked in #2204.

On Windows specifically: flake.nix's eachSystem list only defines x86_64-linux, aarch64-linux, aarch64-darwin — there is no Windows Nix system, and Nix has no native Windows support outside WSL2 (at which point the host is already running a Linux kernel, a different scenario than what this issue targets). A nix develop-based cross-compile workaround (e.g. cargo build --target aarch64-unknown-linux-gnu from inside a macOS or Linux devShell) can partially cover macOS and Linux hosts today, but it does not extend to Windows contributors at all — they have zero local path to this code, cross-compile or otherwise.

Current coverage, by host and check:

Check macOS / Linux host (via nix cross-compile) Windows host
compile mitigated (manual, undocumented) missing
lint likely mitigated, same mechanism missing
test missing — a cross-compiled Linux binary cannot execute on a non-Linux host, regardless of what it exercises missing

Nix closes part of the gap for macOS/Linux hosts but none of it for Windows, and it never closes the test-execution gap anywhere non-Linux. This is why mise run ci:linux / test:linux are still needed, and why the design below treats the container-based mechanism as the required default rather than one of two equally viable options.

Impact / Why This Matters

Without a local Linux path, non-Linux contributors get a green local run, then a red Rust branch-check or merge-queue failure — feedback latency jumps from seconds to a full CI round-trip, and merge-queue failures block other PRs. Today the only workarounds are (a) push and wait for CI, (b) hand-craft a docker run ... cargo ... command that bypasses the repo's docker/podman engine abstraction and mishandles file ownership and build caches, or (c) on macOS/Linux only, use a Nix devShell to cross-compile — which catches compile/lint errors but cannot execute a single test, and has no equivalent at all on Windows. None of these are reproducible, discoverable, or complete, so Linux-gated regressions keep reaching CI. This matters most for the Linux-heavy supervisor/driver/sandbox crates, where the gated surface is largest, and most acutely for Windows contributors, who have no workaround at any level — compile, lint, or test — today.

Proposed Design

New mise tasks, discoverable in mise tasks:

mise run ci:linux     # the ci checks, run for a Linux target in a Linux environment
mise run test:linux   # the test checks, run for a Linux target in a Linux environment

Each realizes a Linux environment, runs the existing ci / test task graph against the host working tree, and exits with the in-environment status. On a Linux host the tasks still work.

Two candidate mechanisms (workflow-level choice for maintainers; see Alternatives):

  • (A) Container-based (recommended default, and the only mechanism that reaches Windows contributors or closes the test-execution gap anywhere). Run the existing task graph inside a container built from the same Dockerfile.ci CI uses. Reproduces target_os = "linux" compile/clippy/test faithfully (identical pinned Rust 1.95.0). It approximates the CI Rust lane rather than being bit-identical — CI's Rust lane uses the Nix devShell, and the mise test:rust task uses cargo test where the CI lane uses cargo nextest run --profile ci — but it catches the class of failure this issue targets. Portable across macOS/Windows/Linux hosts via Docker Desktop or Podman Desktop.
  • (B) Nix-based. Reproduce the CI lane exactly via nix develop .#devShells.x86_64-linux.default. Bit-faithful, but on a non-Linux host requires a Linux Nix builder (remote builder, or a Linux container running the Nix daemon) — heavier prerequisites — and does not work on Windows at all (no Windows Nix system, no native Windows Nix support outside WSL2).

Observable behavior (either mechanism):

  • Runs the repo's standard Linux checks so a local pass predicts the CI Rust lane. mise run ci's rust:format:check and rust:lint already match the CI cargo fmt/cargo clippy invocations verbatim; the test step reuses test:rust (implementers may instead invoke cargo nextest --profile ci to match the CI lane exactly).
  • For the container mechanism, works with docker and podman alike (repo convention).
  • Reuses build caches across runs (cargo registry, target/, sccache); repeat runs are incremental.
  • Host working tree is the source of truth (bind-mounted); artifacts written back do not break subsequent host cargo/git use.
  • Image/toolchain source is explicit, version-pinned, and consistent with mise.toml / Dockerfile.ci.

Essential constraints (existing conventions):

  • Thin task in tasks/ci.toml; logic in a new tasks/scripts/*.sh, mirroring docker-build-ci.sh.
  • For (A): source tasks/scripts/container-engine.sh and use ce run — not bare docker run — so docker and podman both work; keep run flags portable (ce_build normalizes flags but there is no ce_run equivalent).
  • SPDX headers on all new files (enforced by license:check).
  • Handle file ownership for both rootful docker (UID mapping) and rootless podman (userns).
  • Mount cache volumes; account for the global RUSTC_WRAPPER=sccache / SCCACHE_DIR env from mise.toml.
  • ghcr.io/nvidia/openshell/ci is not anonymously pullable (CI authenticates with GITHUB_TOKEN); the task must authenticate to ghcr or fall back to a local build:docker:ci, and document which.
  • Scope: lint + compile + test. Excludes e2e (needs docker-in-docker / a live gateway).

Acceptance Criteria

  • mise run ci:linux runs the ci task graph (format check, clippy -D warnings, tests) for a Linux target in a Linux environment and propagates the exit code.
  • mise run test:linux runs the test task graph for a Linux target in a Linux environment.
  • A deliberately introduced compile error inside a #[cfg(target_os = "linux")] block is caught by mise run ci:linux on a macOS host.
  • The same is verified on a Windows host, since that host has no other local path to this code today.
  • A deliberately introduced test failure inside Linux-gated code is caught by mise run test:linux, verifying the test-execution gap is actually closed (not just compile/lint).
  • For the container mechanism: works with docker and with podman via ce (no bare docker/podman calls).
  • Repeat runs reuse caches (second run substantially faster; no cold rebuild).
  • Artifacts written to the host tree are not left root-owned in a way that breaks subsequent host cargo/git operations.
  • Image/toolchain source is pinned and documented; ghcr-auth vs. local-build behavior is explicit; version-sync with mise.toml/Dockerfile.ci is maintained.
  • New files carry SPDX headers; mise run lint passes.
  • CONTRIBUTING testing docs and the AGENTS.md "Testing" list are updated; CI-referencing skills reviewed per the Skill Maintenance rule / sync-agent-infra.
  • Related issue #2204 linked.

Alternatives Considered

  1. Nix Linux devShell as the only path (design B). Exact parity for compile/lint but needs a Linux Nix builder on non-Linux hosts — a real prerequisite barrier — cannot execute tests on a non-Linux host under any configuration, and provides no path at all for Windows contributors, since Nix has no native Windows target. Kept as the "exact parity" alternative for macOS/Linux compile/lint checks, not the default, and not sufficient on its own.
  2. Cross-compile with cargo-zigbuild (already a dep). Covers compile/clippy but not test execution, and Linux-only runtime behavior (namespaces, Landlock, process supervision) still can't run. Partial, same ceiling as the Nix workaround.
  3. Document a raw docker run snippet in CONTRIBUTING. No engine abstraction, no cache/ownership handling, drifts, undiscoverable. Rejected.
  4. Rely on remote CI feedback only. Status quo; the round-trip latency and merge-queue blocking are the reported problem.
  5. Fold into #2204's unified build mechanism. #2204 is a broad roadmap item; this is a small, independently shippable step. Link as parent rather than block on it.

Agent Investigation

Investigated from a source read of main (refreshed 2026-09-18; original investigation was on a clean working tree two weeks earlier).

  • Counted the Linux-gated surface: grep -rn 'target_os = "linux"' crates/ -> 434 lines, 42 files, 12 crates (up from 382/31/10 at filing) — heaviest by file count: openshell-sandbox, openshell-supervisor-network, openshell-driver-vm, openshell-supervisor-process, openshell-driver-podman.
  • Traced task graph: mise run ci -> lint (rust:format:check, rust:lint), check, test (test:rust = cargo test --workspace --exclude openshell-server + cargo test -p openshell-server --features test-support), go:ci, rust:deny:policy. All host-target.
  • Confirmed CI path: branch-checks.yml Rust lane runs in a Nix devShell (x86_64-linux, aarch64-linux, aarch64-darwin) with cargo fmt/cargo clippy/cargo nextest --profile ci; go/python/license/markdown/cargo-deny run in ghcr.io/nvidia/openshell/ci:latest. build:docker:ci only builds that image; no task runs checks inside a container (grep 'docker run' tasks/ -> none, confirmed still true).
  • Verified conventions: tasks/scripts/container-engine.sh provides the required docker/podman ce/ce_build abstraction (bare docker run would violate it); SPDX headers enforced via license:check; ghcr.io/nvidia/openshell/ci is not anonymously pullable; rust pin (rust-toolchain.toml) still 1.95.0, unchanged since filing.
  • Confirmed flake.nix's eachSystem list (x86_64-linux, aarch64-linux, aarch64-darwin) has no Windows entry — the basis for the Windows-specific note above.
  • Confirmed the test-execution gap is categorical, not a Landlock/namespace-specific edge case: a cross-compiled Linux ELF binary cannot execute on a non-Linux host's kernel at all, so no cargo test/cargo nextest invocation can run under the nix cross-compile workaround regardless of what the test exercises.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.