oxidecomputer / oxidecomputer/omicron

lots of Crucible test failures due to timeouts

Open
#8,738 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I'm running the test suite locally for the first time in a while and ran into a log of test Crucible-related test failures that I'd expect to be unrelated to my changes:

$ cargo nextest run --no-fail-fast
...
────────────
     Summary [1935.375s] 2194 tests run: 2183 passed (95 slow), 11 failed, 3 skipped
        FAIL [  53.948s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_replacement_done
        FAIL [  81.001s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_requested
        FAIL [  53.394s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_replacement_state_running_on_pantry
        FAIL [  53.295s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested
        FAIL [  52.203s] omicron-nexus::test_all integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_running
        FAIL [  51.029s] omicron-nexus::test_all integration_tests::disks::test_disk_expunge
        FAIL [  50.192s] omicron-nexus::test_all integration_tests::disks::test_disk_slot_assignment
        FAIL [  51.146s] omicron-nexus::test_all integration_tests::volume_management::test_delete_image_order_5
        FAIL [  72.444s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_counting
        FAIL [  47.413s] omicron-nexus::test_all integration_tests::volume_management::test_read_only_region_reference_usage_sanity
        FAIL [  11.927s] omicron-omdb::test_all_output test_omdb_usage_errors
error: test run failed

(the omdb one is definitely mine)

This is all on commit ad09008d11a73817571581ce179bc96e1c480fa8 (which is far from "main", just not in ways I'd expect to affect this). I will double check whether this happens in CI and whether I see it on "main" when running locally too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failures with cargo nextest run --no-fail-fast at commit ad09008d11a73817571581ce179bc96e1c480fa8. Inspect the listed integration_tests::crucible_replacements, disks, and volume_management tests separately from test_omdb_usage_errors; done means the Crucible-related timeout failures are explained or fixed and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.