ROCm / ROCm/rocm-cli

E2E: a lane where every @requires-gpu scenario resolves to skip should be flagged, not reported ok

Open
#383 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
40
Forks
9
Avg merge
4d 20h
Merged PRs (30d)
59

Description

Context

Came up during PR #377's review (moving Strix Halo Ubuntu/Windows to the DevLab Dispatch pool). An earlier review pass suggested making the Windows lane's GPU preflight fail-closed (error instead of exit 0) when rocm-smi is absent, to stop a lane silently reporting PASS with zero real GPU scenarios executed. That specific fix is wrong: rocm-smi is absent on the existing static Windows host too (confirmed against the 2026-09-09 nightly log), so a fail-closed preflight would red both Windows lanes on hardware that's demonstrably fine.

The actual gap

The real hole isn't the preflight, it's reconciliation. Confirmed by reading the code:

  • tests/e2e-cucumber/tests/e2e.rs:1225 filters Expectation::Skip out before running scenarios.
  • :1348 only exits 1 on unexpected_fail/stale_xpass.
  • tests/e2e-cucumber/src/expectation.rs:433 resolves @requires-gpu to skip when there's no GPU.
  • crates/e2e-report/src/lib.rs:981 renders pass=0, skip=N as ok().

So a lane whose GPU detection is broken (or that never had a GPU at all) exits 0 and shows green in the consolidated grid — identical to a lane that actually exercised every scenario. Nothing distinguishes "everything passed" from "everything was skipped."

Suggested direction

A portable check that works on any lane, not just Windows: when the capability probe (platform.json) reports the host has an AMD GPU, but every @requires-gpu scenario in that run resolved to skip, treat that as a problem (exit non-zero / flag in the consolidated report) rather than ok. This doesn't depend on rocm-smi specifically or any one platform's tooling, and it catches the actual failure mode (a lane that silently stopped testing anything) without false-positiving on hardware that's fine but lacks a particular diagnostic tool.

Not urgent/blocking anything — a robustness improvement for the E2E harness's reconciliation logic.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/e2e-cucumber/tests/e2e.rs around lines 1225 and 1348, then read tests/e2e-cucumber/src/expectation.rs:433 and crates/e2e-report/src/lib.rs:981. Trace how platform.json capability data and @requires-gpu scenario results are reconciled. Done means a GPU-reported lane with all such scenarios skipped is flagged non-zero or visibly non-ok, while a lane without a GPU remains unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.