oxidecomputer / oxidecomputer/omicron

End-to-end CI test should yield instance serial console output on failure

Open
#4,491 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Testing & Analysis
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Failing run: https://buildomat.eng.oxide.computer/wg/0/details/01HEXKACYWF89VM1E9CTJYVV3R/Lpprn4Zt49bMSVdfGonecyoOgVrADSncQ5GrM2n30Tfe2sK4/01HEXKATT657FMT1H9Z7WPQBJB

The failure stack shows that the test expired here: https://github.com/oxidecomputer/omicron/blob/6d73cf8d856e82883886dade41c956d2fd83034e/end-to-end-tests/src/instance_launch.rs#L113-L142

In other words, the test wanted to see the END SSH HOST KEY KEYS marker on the serial console within 5 minutes and failed when it didn't show up.

There are no Propolis logs here (see #4490), but the sled agent logs show that Propolis at least told sled agent that the guest was running:

SledAgent (InstanceManager): updated state after observing Propolis state change
    file = sled-agent/src/instance.rs:354
    instance_id = f8252e1e-d83a-44b4-b0ef-d218702bc8c9
    new_instance_state = InstanceRuntimeState { propolis_id: Some(a965b005-c60d-43b4-8a01-78fe39d882ed), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-11-10T22:53:25.027228194Z }
    new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2023-11-10T22:53:32.750978136Z }
    propolis_id = a965b005-c60d-43b4-8a01-78fe39d882ed

In situations like this, we'd like to be able to see what the test did see on the serial console so that we can assess where the guest got stuck. (From the virt stack's perspective, control has been handed over to the guest here, and we need to know what it was doing: did it manage to boot into the guest OS at all? If so, what did the guest do after that?)

If this test fails after trying to start the guest, the failure path should dump the guest serial console contents (if any) to an artifact that we can retrieve from the logs or output artifacts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read end-to-end-tests/src/instance_launch.rs around lines 113-142 and inspect the failure path after starting the guest. Run or reproduce the end-to-end CI test, then verify that a failure preserves any guest serial-console contents in a retrievable log or output artifact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.