oxidecomputer / oxidecomputer/omicron

test failed in CI: Several time outs when starting sled agent during volume management tests

Open
#9,972 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on #9967:

https://github.com/oxidecomputer/omicron/pull/9967/checks?check_run_id=65663757734

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KJV88QZMY7XYDA68M86AQDEN/3Z51mBzelMabtysXy2wkNKxpJLrMa5bziwOcBJEeTYtiS5wB/01KJVJ84KQJ24PEY27HE7CSDPN#S8664

https://buildomat.eng.oxide.computer/wg/0/details/01KJV88QZMY7XYDA68M86AQDEN/3Z51mBzelMabtysXy2wkNKxpJLrMa5bziwOcBJEeTYtiS5wB/01KJVJ84KQJ24PEY27HE7CSDPN#S8788

https://buildomat.eng.oxide.computer/wg/0/details/01KJV88QZMY7XYDA68M86AQDEN/3Z51mBzelMabtysXy2wkNKxpJLrMa5bziwOcBJEeTYtiS5wB/01KJVJ84KQJ24PEY27HE7CSDPN#S8965

https://buildomat.eng.oxide.computer/wg/0/details/01KJV88QZMY7XYDA68M86AQDEN/3Z51mBzelMabtysXy2wkNKxpJLrMa5bziwOcBJEeTYtiS5wB/01KJVJ84KQJ24PEY27HE7CSDPN#S9073

Excerpt from the log showing the failure:

thread 'integration_tests::volume_management::test_upstairs_notify_downstairs_client_stop_request' (2) panicked at nexus/src/lib.rs:490:14:
Could not initialize rack: ServiceUnavailable { internal_message: "Failed to access DB connection: Backends exist, and appear online, but all claims are used" }
thread 'integration_tests::volume_management::test_proper_region_sled_redundancy' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/resource_helpers.rs:2096:10:
expected to find inventory collection: TimedOut(400.373382994s)
thread 'integration_tests::volume_management::test_upstairs_notify_downstairs_client_stops' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/starter.rs:241:21:
Timed out after 600s while running step start_sled1
thread 'integration_tests::volume_management::test_upstairs_repair_different_upstairs_retry' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/starter.rs:241:21:
Timed out after 600s while running step start_sled1

At first I was going to add this to https://github.com/oxidecomputer/omicron/issues/8738 but looking a like deeper it seemed that the error was in the simulated sled agent.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked CI run and the four failure excerpts, then read nexus/src/lib.rs, nexus/test-utils/src/resource_helpers.rs, and nexus/test-utils/src/starter.rs at the reported lines. Reproduce the volume-management integration-test failures and trace the simulated sled-agent startup and inventory-collection timeouts. Done means the affected tests complete without the reported initialization or timeout failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.