oxidecomputer / oxidecomputer/omicron
test failed in CI: integration_tests::snapshots::test_snapshot_basic
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on pull request #8529
https://github.com/oxidecomputer/omicron/pull/8529/checks?check_run_id=45499876606
Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01JZK2Q15Y8HGWG7FX7X9VEKWK/6iPsnnn2mbRTJuku47ICKpBcwIOtTJedT0XfXxUSGWIcCiVl/01JZK2QQPDWGEMGMG9YGHHY8J3
Excerpt from the log showing the failure:
7583 2025-07-07T20:05:22.633Z FAIL [ 40.955s] omicron-nexus::test_all integration_tests::snapshots::test_snapshot_basic
7584 2025-07-07T20:05:22.633Z stdout ───
7585 2025-07-07T20:05:22.633Z
7586 2025-07-07T20:05:22.633Z running 1 test
7587 2025-07-07T20:05:22.633Z test integration_tests::snapshots::test_snapshot_basic ... FAILED
7588 2025-07-07T20:05:22.633Z
7589 2025-07-07T20:05:22.633Z failures:
7590 2025-07-07T20:05:22.633Z
7591 2025-07-07T20:05:22.633Z failures:
7592 2025-07-07T20:05:22.633Z integration_tests::snapshots::test_snapshot_basic
7593 2025-07-07T20:05:22.633Z
7594 2025-07-07T20:05:22.633Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 446 filtered out; finished in 40.61s
7595 2025-07-07T20:05:22.633Z
7596 2025-07-07T20:05:22.633Z stderr ───
7597 2025-07-07T20:05:22.633Z log file: /var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.0.log
7598 2025-07-07T20:05:22.633Z note: configured to log to "/var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.0.log"
7599 2025-07-07T20:05:22.633Z DB URL: postgresql://root@[::1]:39847/omicron?sslmode=disable
7600 2025-07-07T20:05:22.633Z DB address: [::1]:39847
7601 2025-07-07T20:05:22.633Z log file: /var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.2.log
7602 2025-07-07T20:05:22.633Z note: configured to log to "/var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.2.log"
7603 2025-07-07T20:05:22.633Z
7604 2025-07-07T20:05:22.633Z thread 'integration_tests::snapshots::test_snapshot_basic' panicked at sled-agent/src/sim/collection.rs:274:65:
7605 2025-07-07T20:05:22.634Z called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:51461/vmms/5b2a6c8f-f90b-450a-bf40-71d169c527c0)" }
7606 2025-07-07T20:05:22.634Z stack backtrace:
7607 2025-07-07T20:05:22.634Z 0: __rustc::rust_begin_unwind
7608 2025-07-07T20:05:22.634Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:697:5
7609 2025-07-07T20:05:22.634Z 1: core::panicking::panic_fmt
7610 2025-07-07T20:05:22.634Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:75:14
7611 2025-07-07T20:05:22.634Z 2: core::result::unwrap_failed
7612 2025-07-07T20:05:22.634Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/result.rs:1732:5
7613 2025-07-07T20:05:22.634Z 3: unwrap<(), omicron_common::api::external::error::Error>
7614 2025-07-07T20:05:22.634Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/result.rs:1137:23
7615 2025-07-07T20:05:22.634Z 4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
7616 2025-07-07T20:05:22.634Z at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:13
7617 2025-07-07T20:05:22.634Z 5: {async_fn#0}
7618 2025-07-07T20:05:22.634Z at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:521:58
7619 2025-07-07T20:05:22.634Z 6: {async_fn#0}
7620 2025-07-07T20:05:22.634Z at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:814:38
7621 2025-07-07T20:05:22.687Z 7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>>
7622 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/handler.rs:664:78
7623 2025-07-07T20:05:22.687Z 8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7624 2025-07-07T20:05:22.687Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7625 2025-07-07T20:05:22.687Z 9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
7626 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/handler.rs:785:16
7627 2025-07-07T20:05:22.687Z 10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7628 2025-07-07T20:05:22.687Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7629 2025-07-07T20:05:22.687Z 11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
7630 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:937:69
7631 2025-07-07T20:05:22.687Z 12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
7632 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331:17
7633 2025-07-07T20:05:22.687Z 13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
7634 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16:9
7635 2025-07-07T20:05:22.687Z 14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
7636 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320:13
7637 2025-07-07T20:05:22.687Z 15: tokio::runtime::task::harness::poll_future::{{closure}}
7638 2025-07-07T20:05:22.687Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532:19
7639 2025-07-07T20:05:22.687Z 16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
7640 2025-07-07T20:05:22.687Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panic/unwind_safe.rs:272:9
7641 2025-07-07T20:05:22.687Z 17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7642 2025-07-07T20:05:22.687Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:589:40
7643 2025-07-07T20:05:22.687Z 18: __rust_try
7644 2025-07-07T20:05:22.687Z 19: try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
7645 2025-07-07T20:05:22.688Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:552:19
7646 2025-07-07T20:05:22.688Z 20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7647 2025-07-07T20:05:22.688Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panic.rs:359:14
7648 2025-07-07T20:05:22.688Z 21: tokio::runtime::task::harness::poll_future
7649 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520:18
7650 2025-07-07T20:05:22.688Z 22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
7651 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209:27
7652 2025-07-07T20:05:22.688Z 23: tokio::runtime::task::harness::Harness<T,S>::poll
7653 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154:15
7654 2025-07-07T20:05:22.688Z 24: tokio::runtime::task::raw::poll
7655 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:271:5
7656 2025-07-07T20:05:22.688Z 25: tokio::runtime::task::raw::RawTask::poll
7657 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:201:18
7658 2025-07-07T20:05:22.688Z 26: tokio::runtime::task::LocalNotified<S>::run
7659 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/mod.rs:463:9
7660 2025-07-07T20:05:22.688Z 27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
7661 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:780:25
7662 2025-07-07T20:05:22.688Z 28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7663 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:167:5
7664 2025-07-07T20:05:22.688Z 29: budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7665 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:133:5
7666 2025-07-07T20:05:22.688Z 30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7667 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:368:43
7668 2025-07-07T20:05:22.688Z 31: tokio::runtime::scheduler::current_thread::Context::enter
7669 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:432:19
7670 2025-07-07T20:05:22.688Z 32: tokio::runtime::scheduler::current_thread::Context::run_task
7671 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:368:23
7672 2025-07-07T20:05:22.688Z 33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7673 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:776:35
7674 2025-07-07T20:05:22.688Z 34: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7675 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:68
7676 2025-07-07T20:05:22.688Z 35: tokio::runtime::context::scoped::Scoped<T>::set
7677 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/scoped.rs:40:9
7678 2025-07-07T20:05:22.688Z 36: tokio::runtime::context::set_scheduler::{{closure}}
7679 2025-07-07T20:05:22.688Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:26
7680 2025-07-07T20:05:22.688Z 37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7681 2025-07-07T20:05:22.688Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:315:12
7682 2025-07-07T20:05:22.688Z 38: std::thread::local::LocalKey<T>::with
7683 2025-07-07T20:05:22.688Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:279:15
7684 2025-07-07T20:05:22.688Z 39: tokio::runtime::context::set_scheduler
7685 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:9
7686 2025-07-07T20:05:22.689Z 40: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7687 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:27
7688 2025-07-07T20:05:22.689Z 41: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7689 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:720:19
7690 2025-07-07T20:05:22.689Z 42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7691 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:200:28
7692 2025-07-07T20:05:22.689Z 43: tokio::runtime::context::runtime::enter_runtime
7693 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/runtime.rs:65:16
7694 2025-07-07T20:05:22.689Z 44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7695 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:188:9
7696 2025-07-07T20:05:22.689Z 45: tokio::runtime::runtime::Runtime::block_on_inner
7697 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:356:47
7698 2025-07-07T20:05:22.689Z 46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7699 2025-07-07T20:05:22.689Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:330:13
7700 2025-07-07T20:05:22.689Z 47: test_snapshot_basic
7701 2025-07-07T20:05:22.689Z at ./tests/integration_tests/snapshots.rs:73:1
7702 2025-07-07T20:05:22.689Z 48: test_all::integration_tests::snapshots::test_snapshot_basic::{{closure}}
7703 2025-07-07T20:05:22.689Z at ./tests/integration_tests/snapshots.rs:74:66
7704 2025-07-07T20:05:22.689Z 49: core::ops::function::FnOnce::call_once
7705 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
7706 2025-07-07T20:05:22.689Z 50: core::ops::function::FnOnce::call_once
7707 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
7708 2025-07-07T20:05:22.689Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7709 2025-07-07T20:05:22.689Z
7710 2025-07-07T20:05:22.689Z thread 'integration_tests::snapshots::test_snapshot_basic' panicked at clients/sled-agent-client/src/lib.rs:367:14:
7711 2025-07-07T20:05:22.689Z instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:44074/vmms/5b2a6c8f-f90b-450a-bf40-71d169c527c0/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
7712 2025-07-07T20:05:22.689Z stack backtrace:
7713 2025-07-07T20:05:22.689Z 0: __rustc::rust_begin_unwind
7714 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:697:5
7715 2025-07-07T20:05:22.689Z 1: core::panicking::panic_fmt
7716 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:75:14
7717 2025-07-07T20:05:22.689Z 2: core::result::unwrap_failed
7718 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/result.rs:1732:5
7719 2025-07-07T20:05:22.689Z 3: core::result::Result<T,E>::expect
7720 2025-07-07T20:05:22.689Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/result.rs:1089:23
7721 2025-07-07T20:05:22.717Z 4: {async_block#0}
7722 2025-07-07T20:05:22.717Z at /work/oxidecomputer/omicron/clients/sled-agent-client/src/lib.rs:363:9
7723 2025-07-07T20:05:22.718Z 5: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>
7724 2025-07-07T20:05:22.718Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7725 2025-07-07T20:05:22.718Z 6: {async_fn#0}
7726 2025-07-07T20:05:22.718Z at ./tests/integration_tests/instances.rs:7193:72
7727 2025-07-07T20:05:22.718Z 7: {async_fn#0}
7728 2025-07-07T20:05:22.718Z at ./tests/integration_tests/snapshots.rs:160:44
7729 2025-07-07T20:05:22.718Z 8: {async_block#0}
7730 2025-07-07T20:05:22.718Z at ./tests/integration_tests/snapshots.rs:73:1
7731 2025-07-07T20:05:22.718Z 9: poll<&mut dyn core::future::future::Future<Output=()>>
7732 2025-07-07T20:05:22.718Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7733 2025-07-07T20:05:22.718Z 10: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7734 2025-07-07T20:05:22.718Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/future/future.rs:124:9
7735 2025-07-07T20:05:22.718Z 11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7736 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:733:54
7737 2025-07-07T20:05:22.718Z 12: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7738 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:167:5
7739 2025-07-07T20:05:22.718Z 13: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7740 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:133:5
7741 2025-07-07T20:05:22.718Z 14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7742 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:733:25
7743 2025-07-07T20:05:22.718Z 15: tokio::runtime::scheduler::current_thread::Context::enter
7744 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:432:19
7745 2025-07-07T20:05:22.718Z 16: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7746 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:732:36
7747 2025-07-07T20:05:22.718Z 17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7748 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:68
7749 2025-07-07T20:05:22.718Z 18: tokio::runtime::context::scoped::Scoped<T>::set
7750 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/scoped.rs:40:9
7751 2025-07-07T20:05:22.718Z 19: tokio::runtime::context::set_scheduler::{{closure}}
7752 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:26
7753 2025-07-07T20:05:22.718Z 20: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7754 2025-07-07T20:05:22.718Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:315:12
7755 2025-07-07T20:05:22.718Z 21: std::thread::local::LocalKey<T>::with
7756 2025-07-07T20:05:22.718Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/thread/local.rs:279:15
7757 2025-07-07T20:05:22.718Z 22: tokio::runtime::context::set_scheduler
7758 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:9
7759 2025-07-07T20:05:22.718Z 23: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7760 2025-07-07T20:05:22.718Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:27
7761 2025-07-07T20:05:22.719Z 24: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7762 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:720:19
7763 2025-07-07T20:05:22.719Z 25: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7764 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:200:28
7765 2025-07-07T20:05:22.719Z 26: tokio::runtime::context::runtime::enter_runtime
7766 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/runtime.rs:65:16
7767 2025-07-07T20:05:22.719Z 27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7768 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:188:9
7769 2025-07-07T20:05:22.719Z 28: tokio::runtime::runtime::Runtime::block_on_inner
7770 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:356:47
7771 2025-07-07T20:05:22.719Z 29: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7772 2025-07-07T20:05:22.719Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:330:13
7773 2025-07-07T20:05:22.719Z 30: test_snapshot_basic
7774 2025-07-07T20:05:22.719Z at ./tests/integration_tests/snapshots.rs:73:1
7775 2025-07-07T20:05:22.719Z 31: test_all::integration_tests::snapshots::test_snapshot_basic::{{closure}}
7776 2025-07-07T20:05:22.719Z at ./tests/integration_tests/snapshots.rs:74:66
7777 2025-07-07T20:05:22.719Z 32: core::ops::function::FnOnce::call_once
7778 2025-07-07T20:05:22.719Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
7779 2025-07-07T20:05:22.719Z 33: core::ops::function::FnOnce::call_once
7780 2025-07-07T20:05:22.719Z at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:250:5
7781 2025-07-07T20:05:22.719Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7782 2025-07-07T20:05:22.719Z WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7783 2025-07-07T20:05:22.719Z WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpebHI4V"
7784 2025-07-07T20:05:22.719Z If you would like to access the database for debugging, run the following:
7785 2025-07-07T20:05:22.719Z
7786 2025-07-07T20:05:22.719Z # Run the database
7787 2025-07-07T20:05:22.719Z cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpebHI4V/data"
7788 2025-07-07T20:05:22.719Z # Access the database. Note the port may change if you run multiple databases.
7789 2025-07-07T20:05:22.719Z cockroach sql --host=localhost:32221 --insecure
7790 2025-07-07T20:05:22.719Z WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 27564) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.1-clickhouse-Knwt4t)
7791 2025-07-07T20:05:22.719Z failed to clean up ClickHouse data dir:
7792 2025-07-07T20:05:22.719Z - /var/tmp/omicron_tmp/test_all-77332df08a327803-test_snapshot_basic.27546.1-clickhouse-Knwt4t: File exists (os error 17)
7793 2025-07-07T20:05:22.719Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7794 2025-07-07T20:05:22.719Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpl42c2g
7795 2025-07-07T20:05:22.719Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7796 2025-07-07T20:05:22.719Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpoavSUs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with integration_tests::snapshots::test_snapshot_basic and the CI log linked in the issue. Read sled-agent/src/sim/collection.rs around line 274, then trace the related calls in sled-agent/src/sim/sled_agent.rs and http_entrypoints.rs. Done means the snapshot integration test completes without the communication-error panic in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100