E2E: a lane where every @requires-gpu scenario resolves to skip should be flagged, not reported ok
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 40
- Forks
- 9
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 59
Description
Context
Came up during PR #377's review (moving Strix Halo Ubuntu/Windows to the DevLab Dispatch pool). An earlier review pass suggested making the Windows lane's GPU preflight fail-closed (error instead of exit 0) when rocm-smi is absent, to stop a lane silently reporting PASS with zero real GPU scenarios executed. That specific fix is wrong: rocm-smi is absent on the existing static Windows host too (confirmed against the 2026-09-09 nightly log), so a fail-closed preflight would red both Windows lanes on hardware that's demonstrably fine.
The actual gap
The real hole isn't the preflight, it's reconciliation. Confirmed by reading the code:
tests/e2e-cucumber/tests/e2e.rs:1225filtersExpectation::Skipout before running scenarios.:1348only exits 1 onunexpected_fail/stale_xpass.tests/e2e-cucumber/src/expectation.rs:433resolves@requires-gputo skip when there's no GPU.crates/e2e-report/src/lib.rs:981renderspass=0, skip=Nasok().
So a lane whose GPU detection is broken (or that never had a GPU at all) exits 0 and shows green in the consolidated grid — identical to a lane that actually exercised every scenario. Nothing distinguishes "everything passed" from "everything was skipped."
Suggested direction
A portable check that works on any lane, not just Windows: when the capability probe (platform.json) reports the host has an AMD GPU, but every @requires-gpu scenario in that run resolved to skip, treat that as a problem (exit non-zero / flag in the consolidated report) rather than ok. This doesn't depend on rocm-smi specifically or any one platform's tooling, and it catches the actual failure mode (a lane that silently stopped testing anything) without false-positiving on hardware that's fine but lacks a particular diagnostic tool.
Not urgent/blocking anything — a robustness improvement for the E2E harness's reconciliation logic.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/e2e-cucumber/tests/e2e.rs around lines 1225 and 1348, then read tests/e2e-cucumber/src/expectation.rs:433 and crates/e2e-report/src/lib.rs:981. Trace how platform.json capability data and @requires-gpu scenario results are reconciled. Done means a GPU-reported lane with all such scenarios skipped is flagged non-zero or visibly non-ok, while a lane without a GPU remains unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100