oxidecomputer / oxidecomputer/omicron
test failed in CI: `app::sagas::instance_start::test::test_action_failure_can_unwind`
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on pull request #10578:
https://github.com/oxidecomputer/omicron/pull/10578/checks?check_run_id=80666755839
Log showing the specific test failure:
Excerpt from the log showing the failure:
7321 2026-06-10T21:35:25.867Z TRY 1 FAIL [ 48.982s] (─────────) omicron-nexus app::sagas::instance_start::test::test_action_failure_can_unwind
7322 2026-06-10T21:35:25.867Z stdout ───
7323 2026-06-10T21:35:25.867Z
7324 2026-06-10T21:35:25.867Z running 1 test
7325 2026-06-10T21:35:25.867Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7632503), kind: System }, version: Some(RouterVersion { router_id: bc9bc2c4-5463-4a24-8985-95e5fd9839fe, version: 3 }), routes: {ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fdbf:e912:f6d3::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fdbf:e912:f6d3::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(70896d62-0cf5-462e-9237-181c5c1a02e5)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(70896d62-0cf5-462e-9237-181c5c1a02e5)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }} }
7326 2026-06-10T21:35:25.867Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7632503), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7327 2026-06-10T21:35:25.867Z test app::sagas::instance_start::test::test_action_failure_can_unwind ... FAILED
7328 2026-06-10T21:35:25.867Z
7329 2026-06-10T21:35:25.867Z failures:
7330 2026-06-10T21:35:25.867Z
7331 2026-06-10T21:35:25.867Z failures:
7332 2026-06-10T21:35:25.867Z app::sagas::instance_start::test::test_action_failure_can_unwind
7333 2026-06-10T21:35:25.867Z
7334 2026-06-10T21:35:25.867Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 311 filtered out; finished in 48.29s
7335 2026-06-10T21:35:25.867Z
7336 2026-06-10T21:35:25.868Z stderr ───
7337 2026-06-10T21:35:25.868Z log file: /var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.0.log
7338 2026-06-10T21:35:25.868Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.0.log"
7339 2026-06-10T21:35:25.868Z DB URL: postgresql://root@[::1]:54443/omicron?sslmode=disable
7340 2026-06-10T21:35:25.868Z DB address: [::1]:54443
7341 2026-06-10T21:35:25.868Z log file: /var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.2.log
7342 2026-06-10T21:35:25.868Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.2.log"
7343 2026-06-10T21:35:25.868Z
7344 2026-06-10T21:35:25.868Z thread 'app::sagas::instance_start::test::test_action_failure_can_unwind' (2) panicked at nexus/src/app/sagas/test_helpers.rs:799:5:
7345 2026-06-10T21:35:25.868Z assertion failed: saga_node_events.is_empty()
7346 2026-06-10T21:35:25.868Z stack backtrace:
7347 2026-06-10T21:35:25.868Z 0: __rustc::rust_begin_unwind
7348 2026-06-10T21:35:25.868Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
7349 2026-06-10T21:35:25.868Z 1: core::panicking::panic_fmt
7350 2026-06-10T21:35:25.868Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
7351 2026-06-10T21:35:25.868Z 2: core::panicking::panic
7352 2026-06-10T21:35:25.868Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:150:5
7353 2026-06-10T21:35:25.868Z 3: {async_fn#0}
7354 2026-06-10T21:35:25.868Z at ./src/app/sagas/test_helpers.rs:799:5
7355 2026-06-10T21:35:25.868Z 4: {async_fn#0}<omicron_nexus::app::sagas::instance_start::SagaInstanceStart, omicron_nexus::app::sagas::instance_start::test::test_action_failure_can_unwind::{async_block#0}::test_action_failure_can_unwind::{async_fn#0}::{closure_env#0}, omicron_nexus::app::sagas::instance_start::test::test_action_failure_can_unwind::{async_block#0}::test_action_failure_can_unwind::{async_fn#0}::{closure_env#1}>
7356 2026-06-10T21:35:25.868Z at ./src/app/sagas/test_helpers.rs:647:57
7357 2026-06-10T21:35:25.868Z 5: {async_fn#0}
7358 2026-06-10T21:35:25.868Z at ./src/app/sagas/instance_start.rs:1622:11
7359 2026-06-10T21:35:25.869Z 6: {async_block#0}
7360 2026-06-10T21:35:25.869Z at ./src/app/sagas/instance_start.rs:1565:5
7361 2026-06-10T21:35:25.869Z 7: poll<&mut dyn core::future::future::Future<Output=()>>
7362 2026-06-10T21:35:25.869Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
7363 2026-06-10T21:35:25.869Z 8: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7364 2026-06-10T21:35:25.869Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
7365 2026-06-10T21:35:25.869Z 9: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7366 2026-06-10T21:35:25.869Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:70
7367 2026-06-10T21:35:25.869Z 10: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7368 2026-06-10T21:35:25.869Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
7369 2026-06-10T21:35:25.869Z 11: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7370 2026-06-10T21:35:25.869Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
7371 2026-06-10T21:35:25.869Z 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7372 2026-06-10T21:35:25.869Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:25
7373 2026-06-10T21:35:25.869Z 13: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#0}>
7374 2026-06-10T21:35:25.869Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:451:19
7375 2026-06-10T21:35:25.870Z 14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7376 2026-06-10T21:35:25.870Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:777:44
7377 2026-06-10T21:35:25.870Z 15: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}
7378 2026-06-10T21:35:25.870Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:68
7379 2026-06-10T21:35:25.870Z 16: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
7380 2026-06-10T21:35:25.870Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/scoped.rs:40:9
7381 2026-06-10T21:35:25.871Z 17: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
7382 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:38
7383 2026-06-10T21:35:25.871Z 18: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7384 2026-06-10T21:35:25.871Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:513:12
7385 2026-06-10T21:35:25.871Z 19: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
7386 2026-06-10T21:35:25.871Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:477:20
7387 2026-06-10T21:35:25.871Z 20: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
7388 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:17
7389 2026-06-10T21:35:25.871Z 21: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>
7390 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:27
7391 2026-06-10T21:35:25.871Z 22: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>
7392 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:765:24
7393 2026-06-10T21:35:25.871Z 23: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7394 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:205:33
7395 2026-06-10T21:35:25.871Z 24: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
7396 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
7397 2026-06-10T21:35:25.871Z 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7398 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:193:9
7399 2026-06-10T21:35:25.871Z 26: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
7400 2026-06-10T21:35:25.871Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:371:52
7401 2026-06-10T21:35:25.872Z 27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7402 2026-06-10T21:35:25.872Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
7403 2026-06-10T21:35:25.872Z 28: test_action_failure_can_unwind
7404 2026-06-10T21:35:25.872Z at ./src/app/sagas/instance_start.rs:1565:5
7405 2026-06-10T21:35:25.872Z 29: omicron_nexus::app::sagas::instance_start::test::test_action_failure_can_unwind::{closure#0}
7406 2026-06-10T21:35:25.872Z at ./src/app/sagas/instance_start.rs:1568:6
7407 2026-06-10T21:35:25.872Z 30: <omicron_nexus::app::sagas::instance_start::test::test_action_failure_can_unwind::{closure#0} as core::ops::function::FnOnce<()>>::call_once
7408 2026-06-10T21:35:25.872Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
7409 2026-06-10T21:35:25.872Z 31: core::ops::function::FnOnce::call_once
7410 2026-06-10T21:35:25.872Z at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
7411 2026-06-10T21:35:25.872Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7412 2026-06-10T21:35:25.872Z WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7413 2026-06-10T21:35:25.872Z WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpuTclC4"
7414 2026-06-10T21:35:25.872Z If you would like to access the database for debugging, run the following:
7415 2026-06-10T21:35:25.927Z
7416 2026-06-10T21:35:25.927Z # Run the database
7417 2026-06-10T21:35:25.927Z cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpuTclC4/data"
7418 2026-06-10T21:35:25.927Z # Access the database. Note the port may change if you run multiple databases.
7419 2026-06-10T21:35:25.928Z cockroach sql --host=localhost:32221 --insecure
7420 2026-06-10T21:35:25.928Z WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 10490) and a temporary directory leaked, /var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.1-clickhouse-PnyOty)
7421 2026-06-10T21:35:25.928Z failed to clean up ClickHouse data dir:
7422 2026-06-10T21:35:25.928Z - /var/tmp/omicron_tmp/omicron_nexus-2530918be1b94115-test_action_failure_can_unwind.10292.1-clickhouse-PnyOty: File exists (os error 17)
7423 2026-06-10T21:35:25.928Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7424 2026-06-10T21:35:25.928Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpFN7iSv
7425 2026-06-10T21:35:25.928Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7426 2026-06-10T21:35:25.928Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpYZ944R
7427 2026-06-10T21:35:25.928Z
Note that we can see a Nextest retry succeed a little bit later, so it's definitely flaky.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with app::sagas::instance_start::test::test_action_failure_can_unwind in nexus/src/app/sagas/instance_start.rs, then inspect the failure path through nexus/src/app/sagas/test_helpers.rs at lines 647 and 799. Reproduce the named test and review its log. Done means the test passes without the saga_node_events assertion failing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100