oxidecomputer / oxidecomputer/propolis
phd: could assert expected number of passed/failed/skipped
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 270
- Forks
- 42
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 6
Description
phd-tests/runner/src/main.rs is where we produce the nice test result: ... X passed; Y failed; Z skipped; W not run; finished in ... line, but actual numbers of pass/fail/skipped vary by guest fixture.
while one would hope there are no failures, some tests involve Linux-isms that don't survive contact with cmd.exe and fail for boring reasons, while other tests skip on guest OSes that don't provide all the desired test facilities (usually some tool not present on Alpine like lshw). finally, at least the CPU topology test is (currently) Intel-only and Linux-only.
it'd be nice for phd-runner to, when running all tests, assert that the number of pass/fail/skip actually match our expectations for the given test image adapter, hardware, etc. if a test moves from passing to skipped without us knowing, that's bad!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in phd-tests/runner/src/main.rs, where phd-runner produces the test-result summary, and trace how running all tests identifies the test image adapter and hardware. Define expected pass, fail, and skip counts for those environments, including the stated Linux and Intel-only cases. Done means a change in counts causes the all-tests run to fail rather than going unnoticed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100