oxidecomputer / oxidecomputer/propolis

phd: could assert expected number of passed/failed/skipped

Open
#1,085 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

testing
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

phd-tests/runner/src/main.rs is where we produce the nice test result: ... X passed; Y failed; Z skipped; W not run; finished in ... line, but actual numbers of pass/fail/skipped vary by guest fixture.

while one would hope there are no failures, some tests involve Linux-isms that don't survive contact with cmd.exe and fail for boring reasons, while other tests skip on guest OSes that don't provide all the desired test facilities (usually some tool not present on Alpine like lshw). finally, at least the CPU topology test is (currently) Intel-only and Linux-only.

it'd be nice for phd-runner to, when running all tests, assert that the number of pass/fail/skip actually match our expectations for the given test image adapter, hardware, etc. if a test moves from passing to skipped without us knowing, that's bad!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in phd-tests/runner/src/main.rs, where phd-runner produces the test-result summary, and trace how running all tests identifies the test image adapter and hardware. Define expected pass, fail, and skip counts for those environments, including the stated Linux and Intel-only cases. Done means a change in counts causes the all-tests run to fail rather than going unnoticed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.