oxidecomputer / oxidecomputer/omicron
test failed in CI: test_delete_volume_region_snapshot_replacement_state_requested_3
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/8923:
https://github.com/oxidecomputer/omicron/pull/8923/checks?check_run_id=49021845698
Log showing the specific test failure:
Excerpt from the log showing the failure:
7477 2025-08-27T17:55:50.908Z stdout ───
7478 2025-08-27T17:55:50.908Z
7479 2025-08-27T17:55:50.908Z running 1 test
7480 2025-08-27T17:55:50.908Z test integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3 has been running for over 60 seconds
7481 2025-08-27T17:55:50.908Z test integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3 ... FAILED
7482 2025-08-27T17:55:50.908Z
7483 2025-08-27T17:55:50.908Z failures:
7484 2025-08-27T17:55:50.908Z
7485 2025-08-27T17:55:50.908Z failures:
7486 2025-08-27T17:55:50.908Z integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3
7487 2025-08-27T17:55:50.908Z
7488 2025-08-27T17:55:50.908Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 456 filtered out; finished in 85.17s
7489 2025-08-27T17:55:50.909Z
7490 2025-08-27T17:55:50.909Z stderr ───
7491 2025-08-27T17:55:50.909Z log file: /var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.0.log
7492 2025-08-27T17:55:50.909Z note: configured to log to "/var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.0.log"
7493 2025-08-27T17:55:50.909Z DB URL: postgresql://root@[::1]:44661/omicron?sslmode=disable
7494 2025-08-27T17:55:50.909Z DB address: [::1]:44661
7495 2025-08-27T17:55:50.909Z log file: /var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.2.log
7496 2025-08-27T17:55:50.909Z note: configured to log to "/var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.2.log"
7497 2025-08-27T17:55:50.910Z log file: /var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.3.log
7498 2025-08-27T17:55:50.910Z note: configured to log to "/var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.3.log"
7499 2025-08-27T17:55:50.910Z [Volume { identity: VolumeIdentity { id: 7d767c15-943b-4ce1-8064-99726213c7bd (volume), time_created: 2025-08-27T17:55:27.501306Z, time_modified: 2025-08-27T17:55:27.501306Z }, time_deleted: Some(2025-08-27T17:55:41.155695Z), rcgen: Generation(Generation(1)), data: "{\"type\":\"volume\",\"id\":\"db483749-0b6a-4ff1-aa32-c566448f7699\",\"block_size\":512,\"sub_volumes\":[{\"type\":\"region\",\"block_size\":512,\"blocks_per_extent\":131072,\"extent_count\":16,\"opts\":{\"id\":\"e11fa5a9-d4a1-4bf3-ab89-1d24f6dd612e\",\"target\":[\"[::1]:2002\",\"[::1]:1002\",\"[::1]:3001\"],\"lossy\":false,\"flush_timeout\":null,\"key\":\"yDHxDT/g3ofXXgUauf28hhLsWOZ1+nzbDtKvptN0r0o=\",\"cert_pem\":null,\"key_pem\":null,\"root_cert_pem\":null,\"control\":null,\"read_only\":true},\"gen\":3}],\"read_only_parent\":null}", resources_to_clean_up: Some("{\"V3\":{\"regions\":[],\"region_snapshots\":[]}}") }]
7500 2025-08-27T17:55:50.910Z
7501 2025-08-27T17:55:50.910Z thread 'integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3' panicked at nexus/tests/integration_tests/crucible_replacements.rs:1501:13:
7502 2025-08-27T17:55:50.910Z assertion failed: volumes.is_empty()
7503 2025-08-27T17:55:50.910Z stack backtrace:
7504 2025-08-27T17:55:50.910Z 0: __rustc::rust_begin_unwind
7505 2025-08-27T17:55:50.910Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7506 2025-08-27T17:55:50.910Z 1: core::panicking::panic_fmt
7507 2025-08-27T17:55:50.910Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7508 2025-08-27T17:55:50.910Z 2: core::panicking::panic
7509 2025-08-27T17:55:50.910Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:145:5
7510 2025-08-27T17:55:50.910Z 3: {async_fn#0}
7511 2025-08-27T17:55:50.910Z at ./tests/integration_tests/crucible_replacements.rs:1501:13
7512 2025-08-27T17:55:50.910Z 4: {async_fn#0}
7513 2025-08-27T17:55:50.910Z at ./tests/integration_tests/crucible_replacements.rs:1785:32
7514 2025-08-27T17:55:50.911Z 5: {async_block#0}
7515 2025-08-27T17:55:50.911Z at ./tests/integration_tests/crucible_replacements.rs:1768:1
7516 2025-08-27T17:55:50.911Z 6: poll<&mut dyn core::future::future::Future<Output=()>>
7517 2025-08-27T17:55:50.911Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7518 2025-08-27T17:55:50.911Z 7: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7519 2025-08-27T17:55:50.911Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7520 2025-08-27T17:55:50.911Z 8: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7521 2025-08-27T17:55:50.911Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:70
7522 2025-08-27T17:55:50.911Z 9: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7523 2025-08-27T17:55:50.911Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
7524 2025-08-27T17:55:50.911Z 10: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7525 2025-08-27T17:55:50.911Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
7526 2025-08-27T17:55:50.913Z 11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7527 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:25
7528 2025-08-27T17:55:50.939Z 12: tokio::runtime::scheduler::current_thread::Context::enter
7529 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
7530 2025-08-27T17:55:50.939Z 13: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7531 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:741:44
7532 2025-08-27T17:55:50.939Z 14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7533 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
7534 2025-08-27T17:55:50.939Z 15: tokio::runtime::context::scoped::Scoped<T>::set
7535 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
7536 2025-08-27T17:55:50.939Z 16: tokio::runtime::context::set_scheduler::{{closure}}
7537 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
7538 2025-08-27T17:55:50.939Z 17: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7539 2025-08-27T17:55:50.939Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
7540 2025-08-27T17:55:50.939Z 18: std::thread::local::LocalKey<T>::with
7541 2025-08-27T17:55:50.939Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
7542 2025-08-27T17:55:50.939Z 19: tokio::runtime::context::set_scheduler
7543 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
7544 2025-08-27T17:55:50.939Z 20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7545 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
7546 2025-08-27T17:55:50.939Z 21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7547 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
7548 2025-08-27T17:55:50.939Z 22: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7549 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
7550 2025-08-27T17:55:50.939Z 23: tokio::runtime::context::runtime::enter_runtime
7551 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
7552 2025-08-27T17:55:50.939Z 24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7553 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
7554 2025-08-27T17:55:50.939Z 25: tokio::runtime::runtime::Runtime::block_on_inner
7555 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
7556 2025-08-27T17:55:50.939Z 26: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7557 2025-08-27T17:55:50.939Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
7558 2025-08-27T17:55:50.939Z 27: test_delete_volume_region_snapshot_replacement_state_requested_3
7559 2025-08-27T17:55:50.940Z at ./tests/integration_tests/crucible_replacements.rs:1768:1
7560 2025-08-27T17:55:50.940Z 28: test_all::integration_tests::crucible_replacements::test_delete_volume_region_snapshot_replacement_state_requested_3::{{closure}}
7561 2025-08-27T17:55:50.940Z at ./tests/integration_tests/crucible_replacements.rs:1771:2
7562 2025-08-27T17:55:50.940Z 29: core::ops::function::FnOnce::call_once
7563 2025-08-27T17:55:50.940Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7564 2025-08-27T17:55:50.940Z 30: core::ops::function::FnOnce::call_once
7565 2025-08-27T17:55:50.940Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7566 2025-08-27T17:55:50.940Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7567 2025-08-27T17:55:50.940Z WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7568 2025-08-27T17:55:50.941Z WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpNgJ6Tg"
7569 2025-08-27T17:55:50.941Z If you would like to access the database for debugging, run the following:
7570 2025-08-27T17:55:50.941Z
7571 2025-08-27T17:55:50.941Z # Run the database
7572 2025-08-27T17:55:50.941Z cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpNgJ6Tg/data"
7573 2025-08-27T17:55:50.941Z # Access the database. Note the port may change if you run multiple databases.
7574 2025-08-27T17:55:50.941Z cockroach sql --host=localhost:32221 --insecure
7575 2025-08-27T17:55:50.941Z WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 22831) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.1-clickhouse-jHKiKO)
7576 2025-08-27T17:55:50.941Z failed to clean up ClickHouse data dir:
7577 2025-08-27T17:55:50.941Z - /var/tmp/omicron_tmp/test_all-916b4bf438c2e923-test_delete_volume_region_snapshot_replacement_state_requested_3.22822.1-clickhouse-jHKiKO: File exists (os error 17)
7578 2025-08-27T17:55:50.941Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7579 2025-08-27T17:55:50.942Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpaoBdn1
7580 2025-08-27T17:55:50.942Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7581 2025-08-27T17:55:50.942Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmp9RkRSu
7582 2025-08-27T17:55:50.942Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7583 2025-08-27T17:55:50.942Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpxiLBXi
7584 2025-08-27T17:55:50.942Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7585 2025-08-27T17:55:50.942Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpIEPhA2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/integration_tests/crucible_replacements.rs, especially test_delete_volume_region_snapshot_replacement_state_requested_3 and the assertion at line 1501. Review the linked CI run and failure log to determine why the volume remains, then verify the test completes successfully with the expected empty volume list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100