oxidecomputer / oxidecomputer/propolis

phd: lspci_lifecycle_test passes incorrectly

Open
#792 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

testing
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

disclaimer: this is because of a framework bug, not a Propolis bug. i noticed that lspci_lifecycle_test fails on an Ubuntu 22.04 guest image i'd put together, and at first thought it was wrong, but the truth is stranger than fiction...

in lspci_lifecycle_test we run both lspci and lshw:
https://github.com/oxidecomputer/propolis/blob/93ed767388c4fab11af8a98ad33fdbeac4098b0c/phd-tests/tests/src/hw.rs#L21-L29

on an Ubuntu guest, the lshw assert fails because before and after messages don't match. the difference in the (rather large) strings of output is only that the machine's serial does not match after being stopped and started. i double-checked on a real instance, and a Debian 11 guest's observed value for serial is in fact the instance's ID, and that ID is stable across a stop and start. again, test bug not real bug.

the immediate issue in the test framework is that in the test we validate that lshw and lspci agree across a StopAndStart, but that action involves spawning a successor VM which makes a new TestVm and in turn gets a new id.

  • we probably should just keep the same TestVm if all we're doing is a StopAndStart. seems like this is the only test using StopAndStart, so that's simple enough.

why in the world did this pass with Alpine or Debian images though? i'm glad you've asked!

# on Alpine:
localhost:~# lshw
-ash: lshw: not found
localhost:~#

# or on a different Alpine:
localhost:~# sudo lspci -vvx
-sh: sudo: not found
localhost:~#

# on Debian:
root@debian:~# lshw
-bash: lshw: command not found
root@debian:~#

assert_eq!("-ash: lshw: not found", "-ash: lshw: not found") or equivalent error from bash will pass every time :) i only have an lshw out-of-the-box on this Ubuntu image, which seems to be why it only fails there.

  • run_shell_command ought to check that the shell command that was run returned 0, and force test authors to deal with unexpected test command failures

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with phd-tests/tests/src/hw.rs, then trace StopAndStart in phd-tests/framework/src/lifecycle.rs and TestVm creation in phd-tests/framework/src/lib.rs and test_vm/mod.rs. Check how run_shell_command handles command status. Done means StopAndStart preserves the expected VM identity and unexpected shell-command failures cannot make the assertions pass silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.