oxidecomputer / oxidecomputer/propolis
phd: lspci_lifecycle_test passes incorrectly
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 270
- Forks
- 42
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 6
Description
disclaimer: this is because of a framework bug, not a Propolis bug. i noticed that lspci_lifecycle_test fails on an Ubuntu 22.04 guest image i'd put together, and at first thought it was wrong, but the truth is stranger than fiction...
in lspci_lifecycle_test we run both lspci and lshw:
https://github.com/oxidecomputer/propolis/blob/93ed767388c4fab11af8a98ad33fdbeac4098b0c/phd-tests/tests/src/hw.rs#L21-L29
on an Ubuntu guest, the lshw assert fails because before and after messages don't match. the difference in the (rather large) strings of output is only that the machine's serial does not match after being stopped and started. i double-checked on a real instance, and a Debian 11 guest's observed value for serial is in fact the instance's ID, and that ID is stable across a stop and start. again, test bug not real bug.
the immediate issue in the test framework is that in the test we validate that lshw and lspci agree across a StopAndStart, but that action involves spawning a successor VM which makes a new TestVm and in turn gets a new id.
- we probably should just keep the same
TestVmif all we're doing is aStopAndStart. seems like this is the only test usingStopAndStart, so that's simple enough.
why in the world did this pass with Alpine or Debian images though? i'm glad you've asked!
# on Alpine:
localhost:~# lshw
-ash: lshw: not found
localhost:~#
# or on a different Alpine:
localhost:~# sudo lspci -vvx
-sh: sudo: not found
localhost:~#
# on Debian:
root@debian:~# lshw
-bash: lshw: command not found
root@debian:~#
assert_eq!("-ash: lshw: not found", "-ash: lshw: not found") or equivalent error from bash will pass every time :) i only have an lshw out-of-the-box on this Ubuntu image, which seems to be why it only fails there.
-
run_shell_commandought to check that the shell command that was run returned 0, and force test authors to deal with unexpected test command failures
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with phd-tests/tests/src/hw.rs, then trace StopAndStart in phd-tests/framework/src/lifecycle.rs and TestVm creation in phd-tests/framework/src/lib.rs and test_vm/mod.rs. Check how run_shell_command handles command status. Done means StopAndStart preserves the expected VM identity and unexpected shell-command failures cannot make the assertions pass silently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100