Remove the e2e-test-hooks feature

Open
#349 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, rust

Research direction

Start with engines/lemonade/src/lib.rs, apps/rocm/src/main.rs, xtask/src/e2e.rs, the workflow files, and the serve-18 scenario in tests/e2e-cucumber. Read AGENTS.md section 9 and inspect the existing contract tests before deciding how the planted fake lemond/lemonade pair should work. Done means serve-18 passes on a GPU lane and all feature declarations, seams, build flags, and feature-specific contract tests are removed.

Written by the indexing model from the issue text.

Description

Problem

e2e-test-hooks is a cargo feature that compiles two scripted-failure seams into rocm:

  • engines/lemonade/src/lib.rs:435 — makes the lemonade backend install fail on demand (with the env-var name declared at :43)
  • apps/rocm/src/main.rs:5005scripted_backend_failure, which lets the no-GPU pre-flight be bypassed so the run can reach the install phase at all

It exists to serve one scenario: @id:serve-lemonade-preparation-recovery (serve-18), which is tagged @requires-no-gpu and therefore runs on the blocking every-PR lane.

The cost of that one scenario is spread across the repo:

site count
feature declarations (apps/rocm, engines/lemonade Cargo.toml) 2
#[cfg] / cfg! seams 3
--features rocm/e2e-test-hooks in workflows (nightly.yml x5, e2e-selfhosted.yml x5) 10
xtask/src/e2e.rs:96 build flag 1
contract tests asserting a lane must be hook-ful 2
contract test asserting a lane must be hook-free 1
defensive assertion in tests/e2e-cucumber/tests/e2e/serving_steps.rs:946-966 1

Two consequences follow from the feature existing at all.

Binaries under test are not the binaries shipped. Every lane that builds with the feature packages and installs a binary that differs from a release build. #342 fixed this for the Windows lifecycle lane. It is still true on Linux: build-and-test runs cargo build --workspace in the dev profile only, so the release build happens inside cargo xtask e2e — with the feature — and the Linux lifecycle lane installs a hook-carrying binary through the real installer.

The contract tests encode the asymmetry instead of removing it. One test asserts the Windows lane must be hook-free, two assert self-hosted and nightly must be hook-ful, and ci.yml's Linux lane is asserted by neither. That gap is the same one that hid the duplicate-build drift #342 fixed, one lane over. Adding a fourth contract test would make the asymmetry permanent rather than fix it.

Proposal

Delete the feature. The enabling step is to stop requiring serve-18 to run on a GPU-less lane:

  1. Retag serve-18 from @requires-no-gpu to @requires-gpu.
  2. Drive its failure with a planted fake lemond/lemonade pair plus a hand-written install manifest, instead of a compiled-in seam. This is viable because resolve_runtime() (engines/lemonade/src/lib.rs:1294-1304) checks only read_manifest() and manifest.lemond.is_file() — it does not verify a SHA, version, or signature — and serve_http() reaches ensure_best_llamacpp_backend at :626 without going through prepare_embeddable, so the SHA-pinned archive download is never on this path.
  3. Delete both seams, the feature declarations, the 11 build-flag sites, and the three contract tests, which all become statements about a feature that no longer exists.

Approaches already ruled out, so they are not re-proposed:

  • Mock the failing request. The install failure boundary is a subprocess (run_lemonade_backend_install, :2786), not HTTP. MockServer mocks a layer that does not exist yet at that point in the run.
  • Prewarm the cache on hosted lanes. $RUNNER_WORKSPACE is ephemeral on hosted runners, and on the prewarmed self-hosted lanes xtask e2e-prewarm already ran rocm engines install, so there is nothing left to fail.
  • Drop the network (e.g. sandlock). Wrong layer on hosted runners — the archive download fails first, before the backend install — and skipped entirely on prewarmed lanes.
  • Mock verify_sha256. Any such mechanism is itself a seam, and it moves the failure mode from fail-closed to fail-open inside the supply-chain verification path. See AGENTS.md section 9.

Tradeoff

Serve-18 loses every-PR coverage and moves to a gated lane. That is a real cost and the reason the seams were written in the first place; it should be an explicit maintainer decision, not a side effect of a cleanup. The counter-argument is that the coverage it currently provides on the every-PR lane is coverage of a code path that only exists in test builds, and that the scenario is rare enough that gated-lane latency is acceptable.

Out of scope

  • Sharing built binaries across the Linux build-and-test and E2E tests jobs (~1m25 each). Pure perf, independent of this feature.
  • #125 (Windows sccache, 1299s -> 418s). Separately approved and mergeable; still the largest available build-time win and needs no new engineering.

Follow-up to #342, which raised this in review.

Dominant language
Rust
Stars
40
Forks
9
Avg merge
4d 20h
Merged PRs (30d)
59

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/rocm-cli

All issues in ROCm/rocm-cli

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.