oxidecomputer / oxidecomputer/omicron
test failed in CI: Several time outs when starting sled agent during volume management tests
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on #9967:
https://github.com/oxidecomputer/omicron/pull/9967/checks?check_run_id=65663757734
Log showing the specific test failure:
Excerpt from the log showing the failure:
thread 'integration_tests::volume_management::test_upstairs_notify_downstairs_client_stop_request' (2) panicked at nexus/src/lib.rs:490:14:
Could not initialize rack: ServiceUnavailable { internal_message: "Failed to access DB connection: Backends exist, and appear online, but all claims are used" }
thread 'integration_tests::volume_management::test_proper_region_sled_redundancy' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/resource_helpers.rs:2096:10:
expected to find inventory collection: TimedOut(400.373382994s)
thread 'integration_tests::volume_management::test_upstairs_notify_downstairs_client_stops' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/starter.rs:241:21:
Timed out after 600s while running step start_sled1
thread 'integration_tests::volume_management::test_upstairs_repair_different_upstairs_retry' (2) panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/starter.rs:241:21:
Timed out after 600s while running step start_sled1
At first I was going to add this to https://github.com/oxidecomputer/omicron/issues/8738 but looking a like deeper it seemed that the error was in the simulated sled agent.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked CI run and the four failure excerpts, then read nexus/src/lib.rs, nexus/test-utils/src/resource_helpers.rs, and nexus/test-utils/src/starter.rs at the reported lines. Reproduce the volume-management integration-test failures and trace the simulated sled-agent startup and inventory-collection timeouts. Done means the affected tests complete without the reported initialization or timeout failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100