oxidecomputer / oxidecomputer/omicron
lots of Crucible test failures due to timeouts
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I'm running the test suite locally for the first time in a while and ran into a log of test Crucible-related test failures that I'd expect to be unrelated to my changes:
$ cargo nextest run --no-fail-fast
...
────────────
Summary [1935.375s] 2194 tests run: 2183 passed (95 slow), 11 failed, 3 skipped
FAIL [ 53.948s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done
FAIL [ 81.001s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
FAIL [ 53.394s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
FAIL [ 53.295s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
FAIL [ 52.203s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running
FAIL [ 51.029s] omicron-nexus::test_all integration_tests::disks::test_disk_expunge
FAIL [ 50.192s] omicron-nexus::test_all integration_tests::disks::test_disk_slot_assignment
FAIL [ 51.146s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_5
FAIL [ 72.444s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
FAIL [ 47.413s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_usage_sanity
FAIL [ 11.927s] omicron-omdb::test_all_output test_omdb_usage_errors
error: test run failed
(the omdb one is definitely mine)
This is all on commit ad09008d11a73817571581ce179bc96e1c480fa8 (which is far from "main", just not in ways I'd expect to affect this). I will double check whether this happens in CI and whether I see it on "main" when running locally too.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failures with cargo nextest run --no-fail-fast at commit ad09008d11a73817571581ce179bc96e1c480fa8. Inspect the listed integration_tests::crucible_replacements, disks, and volume_management tests separately from test_omdb_usage_errors; done means the Crucible-related timeout failures are explained or fixed and the relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100