oxidecomputer / oxidecomputer/omicron

Sagas that abort hang test programs

Open
#808 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Testing sagas often involves failing them in specific ways. However, sagas don't currently handle fallible undo actions very well, just panicking the program. This seems like it should just cause the test program to abort, given that we're using the panic = "abort" disposition when building Omicron. Unfortunately, that's not the case, and we instead see the tests hanging.

To reproduce, comment out this line, and run the test named test_instance_create_saga_removes_instance_database_record.

One should see this test output, showing the hang at the end. 
bnaecker@shale : ~/omicron $ RUST_BACKTRACE=1 PATH=$PATH:$OMICRON_PATH cargo t -- test_instance_create_saga_removes_instance_database_record --nocapture
   Compiling omicron-nexus v0.1.0 (/home/bnaecker/omicron/nexus)
   Compiling nexus-test-utils v0.1.0 (/home/bnaecker/omicron/nexus/test-utils)
    Finished test [unoptimized + debuginfo] target(s) in 1m 20s
     Running unittests (target/debug/deps/db_macros-34a4e570060f6587)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/gateway_client-63787fce03b388f2)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/gateway_messages-b3bd03d5210528eb)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/omicron_common-e4dcd320d3f691b6)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/omicron_gateway-4c391968ee580d0b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/gateway-36e1e9edb9c0611f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_all.rs (target/debug/deps/test_all-2d56d305d6cf7281)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/omicron_nexus-f38e4ee5549063dc)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 71 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/nexus-4dbb9461a28b61b5)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_all.rs (target/debug/deps/test_all-589ada27230ffc66)

running 1 test
log file: "/tmp/test_all-589ada27230ffc66-test_instance_create_saga_removes_instance_database_record.18496.0.log"
note: configured to log to "/tmp/test_all-589ada27230ffc66-test_instance_create_saga_removes_instance_database_record.18496.0.log"
thread 'integration_tests::instances::test_instance_create_saga_removes_instance_database_record' panicked at 'called `Result::unwrap()` on an `Err` value: action failed

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.56/src/error.rs:533:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/result.rs:1911:27
   2: omicron_nexus::sagas::sic_delete_instance_record::{{closure}}
             at ./src/sagas.rs:562:5
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/future/mod.rs:80:19
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/future/future.rs:119:9
   5: steno::saga_exec::SagaExecutor<UserType>::undo_node::{{closure}}
             at /home/bnaecker/.cargo/git/checkouts/steno-adfbaf4765a4f9e1/578498b/src/saga_exec.rs:1014:9
   6: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/future/mod.rs:80:19
   7: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
   8: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
   9: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
  10: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
  11: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/panic/unwind_safe.rs:271:9
  12: std::panicking::try::do_call
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:406:40
  13: __rust_try
  14: std::panicking::try
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:370:19
  15: std::panic::catch_unwind
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panic.rs:133:14
  16: tokio::runtime::task::harness::poll_future
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
  17: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
  18: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
  19: tokio::runtime::task::raw::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  20: tokio::runtime::task::raw::RawTask::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
  21: tokio::runtime::task::LocalNotified<S>::run
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
  22: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:532:25
  23: tokio::coop::with_budget::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
  24: std::thread::local::LocalKey<T>::try_with
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/local.rs:399:16
  25: std::thread::local::LocalKey<T>::with
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/local.rs:375:9
  26: tokio::coop::with_budget
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
  27: tokio::coop::budget
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
  28: tokio::runtime::basic_scheduler::Context::run_task::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:291:29
  29: tokio::runtime::basic_scheduler::Context::enter
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
  30: tokio::runtime::basic_scheduler::Context::run_task
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:291:9
  31: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:531:34
  32: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
  33: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
  34: tokio::runtime::basic_scheduler::CoreGuard::enter
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
  35: tokio::runtime::basic_scheduler::CoreGuard::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
  36: tokio::runtime::basic_scheduler::BasicScheduler::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
  37: tokio::runtime::Runtime::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
  38: test_all::integration_tests::instances::test_instance_create_saga_removes_instance_database_record
             at ./tests/integration_tests/instances.rs:511:1
  39: test_all::integration_tests::instances::test_instance_create_saga_removes_instance_database_record::{{closure}}
             at ./tests/integration_tests/instances.rs:511:1
  40: core::ops::function::FnOnce::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/ops/function.rs:227:5
  41: core::ops::function::FnOnce::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/ops/function.rs:227:5
  42: test::__rust_begin_short_backtrace
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/test/src/lib.rs:585:5
  43: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/alloc/src/boxed.rs:1694:9
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:406:40
  46: std::panicking::try
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:370:19
  47: std::panic::catch_unwind
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panic.rs:133:14
  48: test::run_test_in_process
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/test/src/lib.rs:608:18
  49: test::run_test::run_test_inner::{{closure}}
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/test/src/lib.rs:500:39
  50: test::run_test::run_test_inner::{{closure}}
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/test/src/lib.rs:527:37
  51: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/sys_common/backtrace.rs:123:18
  52: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/mod.rs:484:17
  53: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/panic/unwind_safe.rs:271:9
  54: std::panicking::try::do_call
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:406:40
  55: std::panicking::try
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:370:19
  56: std::panic::catch_unwind
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panic.rs:133:14
  57: std::thread::Builder::spawn_unchecked::{{closure}}
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/mod.rs:483:30
  58: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/ops/function.rs:227:5
  59: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/alloc/src/boxed.rs:1694:9
  60: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/alloc/src/boxed.rs:1694:9
  61: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/sys/unix/thread.rs:106:17
  62: _thrp_setup
  63: <unknown>', /home/bnaecker/.cargo/git/checkouts/steno-adfbaf4765a4f9e1/578498b/src/saga_exec.rs:1014:27
stack backtrace:
   0: rust_begin_unwind
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/panicking.rs:107:14
   2: core::result::unwrap_failed
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/result.rs:1613:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/result.rs:1295:23
   4: steno::saga_exec::SagaExecutor<UserType>::undo_node::{{closure}}
             at /home/bnaecker/.cargo/git/checkouts/steno-adfbaf4765a4f9e1/578498b/src/saga_exec.rs:1014:9
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/future/mod.rs:80:19
   6: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
   7: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
   8: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
   9: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
  10: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/panic/unwind_safe.rs:271:9
  11: std::panicking::try::do_call
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:406:40
  12: __rust_try
  13: std::panicking::try
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panicking.rs:370:19
  14: std::panic::catch_unwind
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/panic.rs:133:14
  15: tokio::runtime::task::harness::poll_future
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
  16: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
  17: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
  18: tokio::runtime::task::raw::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  19: tokio::runtime::task::raw::RawTask::poll
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
  20: tokio::runtime::task::LocalNotified<S>::run
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
  21: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:532:25
  22: tokio::coop::with_budget::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
  23: std::thread::local::LocalKey<T>::try_with
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/local.rs:399:16
  24: std::thread::local::LocalKey<T>::with
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/std/src/thread/local.rs:375:9
  25: tokio::coop::with_budget
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
  26: tokio::coop::budget
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
  27: tokio::runtime::basic_scheduler::Context::run_task::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:291:29
  28: tokio::runtime::basic_scheduler::Context::enter
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
  29: tokio::runtime::basic_scheduler::Context::run_task
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:291:9
  30: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:531:34
  31: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
  32: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
  33: tokio::runtime::basic_scheduler::CoreGuard::enter
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
  34: tokio::runtime::basic_scheduler::CoreGuard::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
  35: tokio::runtime::basic_scheduler::BasicScheduler::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
  36: tokio::runtime::Runtime::block_on
             at /home/bnaecker/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
  37: test_all::integration_tests::instances::test_instance_create_saga_removes_instance_database_record
             at ./tests/integration_tests/instances.rs:511:1
  38: test_all::integration_tests::instances::test_instance_create_saga_removes_instance_database_record::{{closure}}
             at ./tests/integration_tests/instances.rs:511:1
  39: core::ops::function::FnOnce::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/ops/function.rs:227:5
  40: core::ops::function::FnOnce::call_once
             at /rustc/65c55bf931a55e6b1e5ed14ad8623814a7386424/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test integration_tests::instances::test_instance_create_saga_removes_instance_database_record has been running for over 60 seconds

It's clear that the saga execution is aborting, because of the unwrap described in the linked issue. What's not clear is why the whole test is not aborting. With a few printlns inserted, it's clear that the test itself aborts, but we never get to the test-context teardown code. I'm not sure why. There is a thread that appears to be the test runner, and a tokio thread hanging out waiting for ... something. We may need to insert tokio-console here or something similar to understand what tasks are blocking the runtime.

Stack trace when the test executable is blocked 
bnaecker@shale : ~ $ pstack $(pgrep test_all)
18496:  /home/bnaecker/omicron/target/debug/deps/test_all-589ada27230ffc66 tes
--------------------- thread# 1 / lwp# 1 ---------------------
 fffffc7feee811a7 lwp_park (0, 0, 0)
 fffffc7feee7a65b cond_wait_queue (ab81fb0, ab80fa0, 0) + 5b
 fffffc7feee7ac9a __cond_wait (ab81fb0, ab80fa0) + 8a
 fffffc7feee7ad04 cond_wait (ab81fb0, ab80fa0) + 24
 fffffc7feee7ad3b pthread_cond_wait (ab81fb0, ab80fa0) + b
 00000000096a6d11 _ZN3std6thread4park17h8e321917a5f3d6feE () + c1
 00000000096c1355 _ZN3std4sync4mpsc8blocking9WaitToken4wait17h7a484f6528084af3E () + 25
 0000000005b3aa08 _ZN3std4sync4mpsc6shared15Packet$LT$T$GT$4recv17h1b6b50f6bd11dfcdE () + 218
 0000000005b39c11 _ZN3std4sync4mpsc17Receiver$LT$T$GT$4recv17h98dc3b30ccbea05cE () + a1
 0000000005b51e05 _ZN4test7console17run_tests_console17h580c99b768f281b5E () + 2d95
 0000000005b6927f _ZN4test9test_main17h91fdcaaba8a23ffbE () + 10f
 0000000005b6a167 _ZN4test16test_main_static17ha87de61db530a0c2E () + 67
 0000000005a216f5 _ZN8test_all4main17hfbb869286e117392E () + 15
 000000000578b20e _ZN4core3ops8function6FnOnce9call_once17h598f520acbf09422E () + e
 00000000053d46a1 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h0e8c20e2f95675ebE () + 11
 00000000058c5bd4 _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17hf950213bd4fbd74fE () + 14
 00000000096c3258 _ZN3std2rt19lang_start_internal17h52fcf825830000abE () + 208
 00000000058c5bae _ZN3std2rt10lang_start17haa3ba19aa3da341aE () + 2e
 0000000005a2171f main () + 1f
 00000000051a98f7 _start_crt () + 87
 00000000051a9858 _start () + 18
--------------------- thread# 2 / lwp# 2 ---------------------
 fffffc7feee87aba ioctl    (3, d001, fffffc7fee7f1430)
 0000000009599fcb _ZN3mio3sys4unix8selector5epoll8Selector6select17h683c663d27dcd403E () + cb
 00000000095a4f80 _ZN3mio4poll4Poll4poll17hb3468d3d6e95bccfE () + 50
 000000000940a711 _ZN5tokio2io6driver6Driver4turn17h40df08788e1661e2E () + 111
 000000000940b064 _ZN63_$LT$tokio..io..driver..Driver$u20$as$u20$tokio..park..Park$GT$12park_timeout17h56db90922024b949E () + 34
 00000000094e0dc2 _ZN73_$LT$tokio..signal..unix..driver..Driver$u20$as$u20$tokio..park..Park$GT$12park_timeout17h0d861ff73bc53a95E () + 22
 00000000094332f2 _ZN73_$LT$tokio..process..imp..driver..Driver$u20$as$u20$tokio..park..Park$GT$12park_timeout17hd986244400528d79E () + 22
 00000000094a6633 _ZN78_$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$12park_timeout17h8093082e86b13442E () + 53
 000000000941a54f _ZN5tokio4time6driver15Driver$LT$P$GT$12park_timeout17h859655338dec5a22E () + 6f
 00000000094195bd _ZN5tokio4time6driver15Driver$LT$P$GT$13park_internal17h333758aa98934c30E () + 31d
 0000000009419cad _ZN74_$LT$tokio..time..driver..Driver$LT$P$GT$$u20$as$u20$tokio..park..Park$GT$4park17hd408c30139961009E () + 1d
 00000000094a64a7 _ZN78_$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$4park17h7758693f036653f2E () + 37
 00000000094132a8 _ZN68_$LT$tokio..runtime..driver..Driver$u20$as$u20$tokio..park..Park$GT$4park17he48f6a3894e9644eE () + 18
 0000000009410cf8 _ZN5tokio7runtime15basic_scheduler7Context4park28_$u7b$$u7b$closure$u7d$$u7d$17hff3abf8cb470cd38E () + 18
 00000000094117d6 _ZN5tokio7runtime15basic_scheduler7Context5enter17hdb1382dc2e37869aE () + 126
 0000000009410a55 _ZN5tokio7runtime15basic_scheduler7Context4park17hbc4bbd4fb1de360aE () + 215
 000000000567625f _ZN5tokio7runtime15basic_scheduler9CoreGuard8block_on28_$u7b$$u7b$closure$u7d$$u7d$17h43e72e59a4c4ac41E () + 43f
 000000000566f79b _ZN5tokio7runtime15basic_scheduler9CoreGuard5enter28_$u7b$$u7b$closure$u7d$$u7d$17h8499cd1a212f6366E () + 2b
 000000000527e804 _ZN5tokio6macros10scoped_tls18ScopedKey$LT$T$GT$3set17hc45a5a4db1d46400E () + 74
 000000000566bbc6 _ZN5tokio7runtime15basic_scheduler9CoreGuard5enter17h8215b324a989eab3E () + 136
 00000000056707ac _ZN5tokio7runtime15basic_scheduler9CoreGuard8block_on17h9ffa6ef43427c528E () + 3c
 0000000005641355 _ZN5tokio7runtime15basic_scheduler14BasicScheduler8block_on17h8ddbbb089aea96e9E () + 135
 0000000005a15be4 _ZN5tokio7runtime7Runtime8block_on17hc12e4f0950b1008eE () + f4
 00000000053d2ffb _ZN8test_all17integration_tests9instances58test_instance_create_saga_removes_instance_database_record17h1a2cdb62effad2b7E () + 12b
 0000000005394441 _ZN8test_all17integration_tests9instances58test_instance_create_saga_removes_instance_database_record28_$u7b$$u7b$closure$u7d$$u7d$17h43342fa69da68d7bE () + 11
 0000000005787991 _ZN4core3ops8function6FnOnce9call_once17h2163e3fe1dea803bE () + 11
 0000000005b6ff36 _ZN4test28__rust_begin_short_backtrace17hab4876760c14af57E () + 6
 0000000005b6ec7f _ZN4test8run_test14run_test_inner28_$u7b$$u7b$closure$u7d$$u7d$17h3f8ec888ffa63041E () + 2df
 0000000005b3769a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hec69a9c8c727aaddE () + ca
 0000000005b3d435 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17hd9aaba9ffbfac6d4E () + 65
 00000000096d06d7 _ZN3std3sys4unix6thread6Thread3new12thread_start17h35db9a2ae668ff29E () + 27
 fffffc7feee80e4c _thrp_setup (fffffc7feed10240) + 6c
 fffffc7feee81160 _lwp_start ()
--------------------- thread# 3 / lwp# 3 ---------------------
 fffffc7feee811a7 lwp_park (0, 0, 0)
 fffffc7feee7a65b cond_wait_queue (ab81c10, ab80ac0, 0) + 5b
 fffffc7feee7ac9a __cond_wait (ab81c10, ab80ac0) + 8a
 fffffc7feee7ad04 cond_wait (ab81c10, ab80ac0) + 24
 fffffc7feee7ad3b pthread_cond_wait (ab81c10, ab80ac0) + b
 00000000096a6d11 _ZN3std6thread4park17h8e321917a5f3d6feE () + c1
 0000000008a6d26c _ZN17crossbeam_channel7context7Context10wait_until17h02af145743818003E () + 1ac
 0000000008aa6f47 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv28_$u7b$$u7b$closure$u7d$$u7d$17ha3a301d35dfb5e54E () + e7
 0000000008a6db11 _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17he89637f1ec9874c6E () + 61
 0000000008a6d81a _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17h398473e230bf3ebdE () + 8a
 0000000008a78263 _ZN3std6thread5local17LocalKey$LT$T$GT$8try_with17he6c76e636ab754abE () + a3
 0000000008a6d507 _ZN17crossbeam_channel7context7Context4with17h8a9a21068b2299b2E () + 47
 0000000008aa6e14 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv17h50fe7f24e0b1d755E () + 154
 0000000008a86084 _ZN17crossbeam_channel7channel17Receiver$LT$T$GT$4recv17hbe2a2fe3376c8805E () + 204
 0000000008a5ccd5 _ZN10slog_async25AsyncCoreBuilder$LT$D$GT$12spawn_thread28_$u7b$$u7b$closure$u7d$$u7d$17hd37fdc40967dc201E () + 25
 0000000008a862d2 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h417dc613a6b7b7c8E () + 22
 0000000008aa8db2 _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$28_$u7b$$u7b$closure$u7d$$u7d$17h91666910c14c40f7E () + 22
 0000000008a85dc2 _ZN115_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$9call_once17h85e13b6d74d76653E () + 22
 0000000008a865f4 _ZN3std9panicking3try7do_call17hb6a09f93f3b46bdaE () + 44
 0000000008a8c37d __rust_try () + 1d
 0000000008a863cf _ZN3std9panicking3try17h660864801e5c9ad5E () + 7f
 0000000008a6de92 _ZN3std5panic12catch_unwind17ha4188c0ea3d9d82eE () + 22
 0000000008aa8beb _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$17hf72d209e971d7a85E () + 19b
 0000000008aa9e91 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9792877b37e81d2dE () + 11
 00000000096d06d7 _ZN3std3sys4unix6thread6Thread3new12thread_start17h35db9a2ae668ff29E () + 27
 fffffc7feee80e4c _thrp_setup (fffffc7feed10a40) + 6c
 fffffc7feee81160 _lwp_start ()
--------------------- thread# 5 / lwp# 5 ---------------------
 00000000096d06b0 _ZN3std3sys4unix6thread6Thread3new12thread_start17h35db9a2ae668ff29E(), exit value = 0x0000000000000000
        ** zombie (exited, not detached, not yet joined) **
--------------------- thread# 6 / lwp# 6 ---------------------
 fffffc7feee811a7 lwp_park (0, 0, 0)
 fffffc7feee7a65b cond_wait_queue (ae5ded0, af4e370, 0) + 5b
 fffffc7feee7ac9a __cond_wait (ae5ded0, af4e370) + 8a
 fffffc7feee7ad04 cond_wait (ae5ded0, af4e370) + 24
 fffffc7feee7ad3b pthread_cond_wait (ae5ded0, af4e370) + b
 00000000096a6d11 _ZN3std6thread4park17h8e321917a5f3d6feE () + c1
 0000000008a6d26c _ZN17crossbeam_channel7context7Context10wait_until17h02af145743818003E () + 1ac
 0000000008aa6f47 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv28_$u7b$$u7b$closure$u7d$$u7d$17ha3a301d35dfb5e54E () + e7
 0000000008a6db11 _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17he89637f1ec9874c6E () + 61
 0000000008a6d81a _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17h398473e230bf3ebdE () + 8a
 0000000008a78263 _ZN3std6thread5local17LocalKey$LT$T$GT$8try_with17he6c76e636ab754abE () + a3
 0000000008a6d507 _ZN17crossbeam_channel7context7Context4with17h8a9a21068b2299b2E () + 47
 0000000008aa6e14 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv17h50fe7f24e0b1d755E () + 154
 0000000008a86084 _ZN17crossbeam_channel7channel17Receiver$LT$T$GT$4recv17hbe2a2fe3376c8805E () + 204
 0000000008a5cb55 _ZN10slog_async25AsyncCoreBuilder$LT$D$GT$12spawn_thread28_$u7b$$u7b$closure$u7d$$u7d$17h4ec422a122a71c71E () + 25
 0000000008a86325 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h7d9d463451045dfbE () + 25
 0000000008aa8d82 _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$28_$u7b$$u7b$closure$u7d$$u7d$17h4f2253e5fce90c0bE () + 22
 0000000008a85df2 _ZN115_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$9call_once17hb6c07aac2b2f3d26E () + 22
 0000000008a8659a _ZN3std9panicking3try7do_call17ha5cce4561c0a96a1E () + 4a
 0000000008a8c37d __rust_try () + 1d
 0000000008a864d2 _ZN3std9panicking3try17hb56517145b1aa4c2E () + 82
 0000000008a6de55 _ZN3std5panic12catch_unwind17h19c055b95ec2848aE () + 25
 0000000008aa88cd _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$17ha2368fef636b8ec2E () + 1cd
 0000000008aa9eb1 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17ha8c0091ca2481487E () + 11
 00000000096d06d7 _ZN3std3sys4unix6thread6Thread3new12thread_start17h35db9a2ae668ff29E () + 27
 fffffc7feee80e4c _thrp_setup (fffffc7feed12240) + 6c
 fffffc7feee81160 _lwp_start ()
--------------------- thread# 7 / lwp# 7 ---------------------
 fffffc7feee811a7 lwp_park (0, 0, 0)
 fffffc7feee7a65b cond_wait_queue (b1323f0, b045070, 0) + 5b
 fffffc7feee7ac9a __cond_wait (b1323f0, b045070) + 8a
 fffffc7feee7ad04 cond_wait (b1323f0, b045070) + 24
 fffffc7feee7ad3b pthread_cond_wait (b1323f0, b045070) + b
 00000000096a6d11 _ZN3std6thread4park17h8e321917a5f3d6feE () + c1
 0000000008a6d26c _ZN17crossbeam_channel7context7Context10wait_until17h02af145743818003E () + 1ac
 0000000008aa6f47 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv28_$u7b$$u7b$closure$u7d$$u7d$17ha3a301d35dfb5e54E () + e7
 0000000008a6db11 _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17he89637f1ec9874c6E () + 61
 0000000008a6d81a _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17h398473e230bf3ebdE () + 8a
 0000000008a78263 _ZN3std6thread5local17LocalKey$LT$T$GT$8try_with17he6c76e636ab754abE () + a3
 0000000008a6d507 _ZN17crossbeam_channel7context7Context4with17h8a9a21068b2299b2E () + 47
 0000000008aa6e14 _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv17h50fe7f24e0b1d755E () + 154
 0000000008a86084 _ZN17crossbeam_channel7channel17Receiver$LT$T$GT$4recv17hbe2a2fe3376c8805E () + 204
 0000000008a5cb55 _ZN10slog_async25AsyncCoreBuilder$LT$D$GT$12spawn_thread28_$u7b$$u7b$closure$u7d$$u7d$17h4ec422a122a71c71E () + 25
 0000000008a86325 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h7d9d463451045dfbE () + 25
 0000000008aa8d82 _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$28_$u7b$$u7b$closure$u7d$$u7d$17h4f2253e5fce90c0bE () + 22
 0000000008a85df2 _ZN115_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$9call_once17hb6c07aac2b2f3d26E () + 22
 0000000008a8659a _ZN3std9panicking3try7do_call17ha5cce4561c0a96a1E () + 4a
 0000000008a8c37d __rust_try () + 1d
 0000000008a864d2 _ZN3std9panicking3try17hb56517145b1aa4c2E () + 82
 0000000008a6de55 _ZN3std5panic12catch_unwind17h19c055b95ec2848aE () + 25
 0000000008aa88cd _ZN3std6thread7Builder15spawn_unchecked28_$u7b$$u7b$closure$u7d$$u7d$17ha2368fef636b8ec2E () + 1cd
 0000000008aa9eb1 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17ha8c0091ca2481487E () + 11
 00000000096d06d7 _ZN3std3sys4unix6thread6Thread3new12thread_start17h35db9a2ae668ff29E () + 27
 fffffc7feee80e4c _thrp_setup (fffffc7feed12a40) + 6c
 fffffc7feee81160 _lwp_start ()
--------------------- thread# 8 / lwp# 8 ---------------------
 fffffc7feee811a7 lwp_park (0, fffffc7feeabbe60, 0)
 fffffc7feee7a65b cond_wait_queue (fffffc7fef006820, fffffc7fef006840, fffffc7feeabbe60) + 5b
 fffffc7feee7aaad cond_wait_common (fffffc7fef006820, fffffc7fef006840, fffffc7feeabbe60) + 1dd
 fffffc7feee7adb5 __cond_timedwait (fffffc7fef006820, fffffc7fef006840, fffffc7feeabbf50) + 65
 fffffc7feee7ae71 cond_timedwait (fffffc7fef006820, fffffc7fef006840, fffffc7feeabbf50) + 31
 fffffc7feefb9dd6 umem_update_thread (0) + 1c6
 fffffc7feee80e4c _thrp_setup (fffffc7feed13240) + 6c
 fffffc7feee81160 _lwp_start ()

There's also a zombie thread in the middle there, but it has an exitcode of 0, so I don't think it's the aborted saga thread.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the commented line in nexus/src/sagas.rs around line 540 and run test_instance_create_saga_removes_instance_database_record using the cargo test command shown. Read the failure path at nexus/src/sagas.rs:562 and the steno saga_exec.rs undo_node frame around line 1014. Done means the reproduced fallible undo case terminates according to the configured panic behavior instead of hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.