oxidecomputer / oxidecomputer/omicron
test failed in CI: test_racing_replacements_for_soft_deleted_disk_volume
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on #8793:
https://github.com/oxidecomputer/omicron/pull/8793/checks?check_run_id=47633713534
Log showing the specific test failure:
Excerpt from the log showing the failure:
7461 2025-08-07T22:46:37.091Z FAIL [ 75.113s] omicron-nexus::test_all integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume
7462 2025-08-07T22:46:37.091Z stdout ───
7463 2025-08-07T22:46:37.091Z
7464 2025-08-07T22:46:37.091Z running 1 test
7465 2025-08-07T22:46:37.091Z test integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume has been running for over 60 seconds
7466 2025-08-07T22:46:37.091Z test integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume ... FAILED
7467 2025-08-07T22:46:37.091Z
7468 2025-08-07T22:46:37.092Z failures:
7469 2025-08-07T22:46:37.092Z
7470 2025-08-07T22:46:37.092Z failures:
7471 2025-08-07T22:46:37.092Z integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume
7472 2025-08-07T22:46:37.092Z
7473 2025-08-07T22:46:37.092Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 452 filtered out; finished in 74.71s
7474 2025-08-07T22:46:37.092Z
7475 2025-08-07T22:46:37.092Z stderr ───
7476 2025-08-07T22:46:37.092Z log file: /var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.0.log
7477 2025-08-07T22:46:37.092Z note: configured to log to "/var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.0.log"
7478 2025-08-07T22:46:37.092Z DB URL: postgresql://root@[::1]:62634/omicron?sslmode=disable
7479 2025-08-07T22:46:37.092Z DB address: [::1]:62634
7480 2025-08-07T22:46:37.092Z log file: /var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.2.log
7481 2025-08-07T22:46:37.092Z note: configured to log to "/var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.2.log"
7482 2025-08-07T22:46:37.092Z log file: /var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.3.log
7483 2025-08-07T22:46:37.092Z note: configured to log to "/var/tmp/omicron_tmp/test_all-a7445ac8ad43db82-test_racing_replacements_for_soft_deleted_disk_volume.22394.3.log"
7484 2025-08-07T22:46:37.092Z old_region_id: Some(Region { identity: RegionIdentity { id: 77b77b81-81ae-40a1-bf0e-57db31e8143a, time_created: 2025-08-07T22:46:11.598723Z, time_modified: 2025-08-07T22:46:11.598723Z }, dataset_id: 149d20c7-9f4c-4024-be94-7855c1d3e364 (dataset), volume_id: 20f3c7c2-86c9-449f-b342-7bd173f625e5 (volume), block_size: ByteCount(ByteCount(512)), blocks_per_extent: 131072, extent_count: 16, port: Some(SqlU16(4000)), read_only: false, deleting: false, reservation_percent: TwentyFive })
7485 2025-08-07T22:46:37.092Z new region id: None
7486 2025-08-07T22:46:37.092Z
7487 2025-08-07T22:46:37.092Z thread 'integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume' panicked at nexus/tests/integration_tests/crucible_replacements.rs:1074:5:
7488 2025-08-07T22:46:37.092Z assertion failed: match last_background_task.last {
7489 2025-08-07T22:46:37.092Z LastResult::Completed(last_result_completed) => {
7490 2025-08-07T22:46:37.092Z match serde_json::from_value::<RegionReplacementDriverStatus>(last_result_completed.details)
7491 2025-08-07T22:46:37.092Z {
7492 2025-08-07T22:46:37.092Z Err(e) => { eprintln!("{e}"); false }
7493 2025-08-07T22:46:37.092Z Ok(v) => !v.drive_invoked_ok.is_empty(),
7494 2025-08-07T22:46:37.092Z }
7495 2025-08-07T22:46:37.092Z }
7496 2025-08-07T22:46:37.092Z _ => { false }
7497 2025-08-07T22:46:37.092Z }
7498 2025-08-07T22:46:37.092Z stack backtrace:
7499 2025-08-07T22:46:37.092Z 0: __rustc::rust_begin_unwind
7500 2025-08-07T22:46:37.092Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:697:5
7501 2025-08-07T22:46:37.092Z 1: core::panicking::panic_fmt
7502 2025-08-07T22:46:37.092Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:75:14
7503 2025-08-07T22:46:37.092Z 2: core::panicking::panic
7504 2025-08-07T22:46:37.093Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:145:5
7505 2025-08-07T22:46:37.093Z 3: {async_fn#0}
7506 2025-08-07T22:46:37.093Z at ./tests/integration_tests/crucible_replacements.rs:1074:5
7507 2025-08-07T22:46:37.093Z 4: {async_block#0}
7508 2025-08-07T22:46:37.093Z at ./tests/integration_tests/crucible_replacements.rs:730:1
7509 2025-08-07T22:46:37.093Z 5: poll<&mut dyn core::future::future::Future<Output=()>>
7510 2025-08-07T22:46:37.093Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7511 2025-08-07T22:46:37.093Z 6: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7512 2025-08-07T22:46:37.093Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7513 2025-08-07T22:46:37.093Z 7: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7514 2025-08-07T22:46:37.093Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:54
7515 2025-08-07T22:46:37.093Z 8: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7516 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
7517 2025-08-07T22:46:37.094Z 9: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7518 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
7519 2025-08-07T22:46:37.094Z 10: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7520 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:25
7521 2025-08-07T22:46:37.094Z 11: tokio::runtime::scheduler::current_thread::Context::enter
7522 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
7523 2025-08-07T22:46:37.094Z 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7524 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:741:36
7525 2025-08-07T22:46:37.094Z 13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7526 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
7527 2025-08-07T22:46:37.094Z 14: tokio::runtime::context::scoped::Scoped<T>::set
7528 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
7529 2025-08-07T22:46:37.094Z 15: tokio::runtime::context::set_scheduler::{{closure}}
7530 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:26
7531 2025-08-07T22:46:37.094Z 16: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7532 2025-08-07T22:46:37.094Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:315:12
7533 2025-08-07T22:46:37.094Z 17: std::thread::local::LocalKey<T>::with
7534 2025-08-07T22:46:37.094Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:279:15
7535 2025-08-07T22:46:37.094Z 18: tokio::runtime::context::set_scheduler
7536 2025-08-07T22:46:37.094Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:9
7537 2025-08-07T22:46:37.094Z 19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7538 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
7539 2025-08-07T22:46:37.095Z 20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7540 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:19
7541 2025-08-07T22:46:37.095Z 21: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7542 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:28
7543 2025-08-07T22:46:37.095Z 22: tokio::runtime::context::runtime::enter_runtime
7544 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
7545 2025-08-07T22:46:37.095Z 23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7546 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
7547 2025-08-07T22:46:37.095Z 24: tokio::runtime::runtime::Runtime::block_on_inner
7548 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:47
7549 2025-08-07T22:46:37.095Z 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7550 2025-08-07T22:46:37.095Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:13
7551 2025-08-07T22:46:37.095Z 26: test_racing_replacements_for_soft_deleted_disk_volume
7552 2025-08-07T22:46:37.095Z at ./tests/integration_tests/crucible_replacements.rs:730:1
7553 2025-08-07T22:46:37.095Z 27: test_all::integration_tests::crucible_replacements::test_racing_replacements_for_soft_deleted_disk_volume::{{closure}}
7554 2025-08-07T22:46:37.095Z at ./tests/integration_tests/crucible_replacements.rs:733:2
7555 2025-08-07T22:46:37.095Z 28: core::ops::function::FnOnce::call_once
7556 2025-08-07T22:46:37.095Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
7557 2025-08-07T22:46:37.095Z 29: core::ops::function::FnOnce::call_once
7558 2025-08-07T22:46:37.095Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/integration_tests/crucible_replacements.rs, especially test_racing_replacements_for_soft_deleted_disk_volume and the assertion at line 1074. Reproduce the CI failure and inspect the linked build log and background-task result around LastResult::Completed. Done means the test reliably passes in CI and no longer fails during the soft-deleted disk-volume replacement race.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100