test failed in CI: test_instance_failed_when_on_expunged_sled

This test failed on a CI run on pull request 8878:

https://github.com/oxidecomputer/omicron/pull/8878/checks?check_run_id=48857788507

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01K3H9XRPGF7BNGWH4NHQM4Z3S/M8Iq0v43RCGitgB0ciOhKSSAiAKjOjKeCyz5jtHxeHY6EjZF/01K3HK24QWY7HJ615ZDHT5HVQ6

Excerpt from the log showing the failure:

FAIL [  44.068s] omicron-nexus::test_all integration_tests::instances::test_instance_failed_when_on_expunged_sled
7610 | 2025-08-25T23:14:38.949Z | stdout ───
7611 | 2025-08-25T23:14:38.949Z |  
7612 | 2025-08-25T23:14:38.950Z | running 1 test
7613 | 2025-08-25T23:14:38.950Z | sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7847805), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7614 | 2025-08-25T23:14:38.950Z | sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7847805), kind: System }, version: Some(RouterVersion { router_id: 43ffa81c-26a9-44fa-bff0-ae18ef8fd956, version: 3 }), routes: {ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fdd9:117a:b95d::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fdd9:117a:b95d::, width: 64 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(f33b2759-a7df-4874-a7c2-3bd334561f31)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(f33b2759-a7df-4874-a7c2-3bd334561f31)) }} }
7615 | 2025-08-25T23:14:38.950Z | test integration_tests::instances::test_instance_failed_when_on_expunged_sled ... FAILED
7616 | 2025-08-25T23:14:38.950Z |  
7617 | 2025-08-25T23:14:38.950Z | failures:
7618 | 2025-08-25T23:14:38.950Z |  
7619 | 2025-08-25T23:14:38.950Z | failures:
7620 | 2025-08-25T23:14:38.950Z | integration_tests::instances::test_instance_failed_when_on_expunged_sled
7621 | 2025-08-25T23:14:38.950Z |  
7622 | 2025-08-25T23:14:38.950Z | test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 455 filtered out; finished in 43.51s
7623 | 2025-08-25T23:14:38.950Z |  
7624 | 2025-08-25T23:14:38.950Z | stderr ───
7625 | 2025-08-25T23:14:38.950Z | log file: /var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.0.log
7626 | 2025-08-25T23:14:38.950Z | note: configured to log to "/var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.0.log"
7627 | 2025-08-25T23:14:38.950Z | DB URL: postgresql://root@[::1]:41822/omicron?sslmode=disable
7628 | 2025-08-25T23:14:38.950Z | DB address: [::1]:41822
7629 | 2025-08-25T23:14:38.950Z | log file: /var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.2.log
7630 | 2025-08-25T23:14:38.950Z | note: configured to log to "/var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.2.log"
7631 | 2025-08-25T23:14:38.950Z | log file: /var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.3.log
7632 | 2025-08-25T23:14:38.951Z | note: configured to log to "/var/tmp/omicron_tmp/test_all-c82d4437db52e771-test_instance_failed_when_on_expunged_sled.25181.3.log"
7633 | 2025-08-25T23:14:38.951Z |  
7634 | 2025-08-25T23:14:38.951Z | thread 'integration_tests::instances::test_instance_failed_when_on_expunged_sled' panicked at sled-agent/src/sim/collection.rs:274:65:
7635 | 2025-08-25T23:14:38.951Z | called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:53925/vmms/eb833377-4575-4894-a987-ce7d9ca22550)" }
7636 | 2025-08-25T23:14:38.951Z | stack backtrace:
7637 | 2025-08-25T23:14:38.951Z | 0: __rustc::rust_begin_unwind
7638 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7639 | 2025-08-25T23:14:38.951Z | 1: core::panicking::panic_fmt
7640 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7641 | 2025-08-25T23:14:38.951Z | 2: core::result::unwrap_failed
7642 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
7643 | 2025-08-25T23:14:38.951Z | 3: unwrap<(), omicron_common::api::external::error::Error>
7644 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1167:23
7645 | 2025-08-25T23:14:38.951Z | 4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
7646 | 2025-08-25T23:14:38.951Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:65
7647 | 2025-08-25T23:14:38.951Z | 5: {async_fn#0}
7648 | 2025-08-25T23:14:38.951Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:523:58
7649 | 2025-08-25T23:14:38.951Z | 6: {async_fn#0}
7650 | 2025-08-25T23:14:38.951Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:823:38
7651 | 2025-08-25T23:14:38.951Z | 7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>>
7652 | 2025-08-25T23:14:38.951Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.3/src/handler.rs:664:78
7653 | 2025-08-25T23:14:38.951Z | 8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7654 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7655 | 2025-08-25T23:14:38.951Z | 9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
7656 | 2025-08-25T23:14:38.951Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.3/src/handler.rs:793:16
7657 | 2025-08-25T23:14:38.951Z | 10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7658 | 2025-08-25T23:14:38.951Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7659 | 2025-08-25T23:14:38.951Z | 11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
7660 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.3/src/server.rs:937:69
7661 | 2025-08-25T23:14:38.952Z | 12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
7662 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:365:24
7663 | 2025-08-25T23:14:38.952Z | 13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
7664 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/loom/std/unsafe_cell.rs:16:9
7665 | 2025-08-25T23:14:38.952Z | 14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
7666 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:354:30
7667 | 2025-08-25T23:14:38.952Z | 15: tokio::runtime::task::harness::poll_future::{{closure}}
7668 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:535:30
7669 | 2025-08-25T23:14:38.952Z | 16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
7670 | 2025-08-25T23:14:38.952Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panic/unwind_safe.rs:272:9
7671 | 2025-08-25T23:14:38.952Z | 17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7672 | 2025-08-25T23:14:38.952Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:589:40
7673 | 2025-08-25T23:14:38.952Z | 18: __rust_try
7674 | 2025-08-25T23:14:38.952Z | 19: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
7675 | 2025-08-25T23:14:38.952Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:552:19
7676 | 2025-08-25T23:14:38.952Z | 20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7677 | 2025-08-25T23:14:38.952Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panic.rs:359:14
7678 | 2025-08-25T23:14:38.952Z | 21: tokio::runtime::task::harness::poll_future
7679 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:523:18
7680 | 2025-08-25T23:14:38.952Z | 22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
7681 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:210:27
7682 | 2025-08-25T23:14:38.952Z | 23: tokio::runtime::task::harness::Harness<T,S>::poll
7683 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:155:20
7684 | 2025-08-25T23:14:38.952Z | 24: tokio::runtime::task::raw::poll
7685 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:325:13
7686 | 2025-08-25T23:14:38.952Z | 25: tokio::runtime::task::raw::RawTask::poll
7687 | 2025-08-25T23:14:38.952Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:255:18
7688 | 2025-08-25T23:14:38.953Z | 26: tokio::runtime::task::LocalNotified<S>::run
7689 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/mod.rs:509:13
7690 | 2025-08-25T23:14:38.953Z | 27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
7691 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:789:30
7692 | 2025-08-25T23:14:38.953Z | 28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7693 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
7694 | 2025-08-25T23:14:38.953Z | 29: budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7695 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
7696 | 2025-08-25T23:14:38.953Z | 30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7697 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:43
7698 | 2025-08-25T23:14:38.953Z | 31: tokio::runtime::scheduler::current_thread::Context::enter
7699 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
7700 | 2025-08-25T23:14:38.953Z | 32: tokio::runtime::scheduler::current_thread::Context::run_task
7701 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:28
7702 | 2025-08-25T23:14:38.953Z | 33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7703 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:785:43
7704 | 2025-08-25T23:14:38.953Z | 34: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7705 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
7706 | 2025-08-25T23:14:38.953Z | 35: tokio::runtime::context::scoped::Scoped<T>::set
7707 | 2025-08-25T23:14:38.953Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
7708 | 2025-08-25T23:14:38.953Z | 36: tokio::runtime::context::set_scheduler::{{closure}}
7709 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
7710 | 2025-08-25T23:14:38.989Z | 37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7711 | 2025-08-25T23:14:38.989Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
7712 | 2025-08-25T23:14:38.989Z | 38: std::thread::local::LocalKey<T>::with
7713 | 2025-08-25T23:14:38.989Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
7714 | 2025-08-25T23:14:38.989Z | 39: tokio::runtime::context::set_scheduler
7715 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
7716 | 2025-08-25T23:14:38.989Z | 40: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7717 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
7718 | 2025-08-25T23:14:38.989Z | 41: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7719 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
7720 | 2025-08-25T23:14:38.989Z | 42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7721 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
7722 | 2025-08-25T23:14:38.989Z | 43: tokio::runtime::context::runtime::enter_runtime
7723 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
7724 | 2025-08-25T23:14:38.989Z | 44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7725 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
7726 | 2025-08-25T23:14:38.989Z | 45: tokio::runtime::runtime::Runtime::block_on_inner
7727 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
7728 | 2025-08-25T23:14:38.989Z | 46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7729 | 2025-08-25T23:14:38.989Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
7730 | 2025-08-25T23:14:38.990Z | 47: test_instance_failed_when_on_expunged_sled
7731 | 2025-08-25T23:14:38.990Z | at ./tests/integration_tests/instances.rs:1237:1
7732 | 2025-08-25T23:14:38.990Z | 48: test_all::integration_tests::instances::test_instance_failed_when_on_expunged_sled::{{closure}}
7733 | 2025-08-25T23:14:38.990Z | at ./tests/integration_tests/instances.rs:1240:2
7734 | 2025-08-25T23:14:38.990Z | 49: core::ops::function::FnOnce::call_once
7735 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7736 | 2025-08-25T23:14:38.990Z | 50: core::ops::function::FnOnce::call_once
7737 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7738 | 2025-08-25T23:14:38.990Z | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7739 | 2025-08-25T23:14:38.990Z |  
7740 | 2025-08-25T23:14:38.990Z | thread 'integration_tests::instances::test_instance_failed_when_on_expunged_sled' panicked at clients/sled-agent-client/src/lib.rs:369:14:
7741 | 2025-08-25T23:14:38.990Z | instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:46583/vmms/eb833377-4575-4894-a987-ce7d9ca22550/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
7742 | 2025-08-25T23:14:38.990Z | stack backtrace:
7743 | 2025-08-25T23:14:38.990Z | 0: __rustc::rust_begin_unwind
7744 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7745 | 2025-08-25T23:14:38.990Z | 1: core::panicking::panic_fmt
7746 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7747 | 2025-08-25T23:14:38.990Z | 2: core::result::unwrap_failed
7748 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
7749 | 2025-08-25T23:14:38.990Z | 3: core::result::Result<T,E>::expect
7750 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1119:23
7751 | 2025-08-25T23:14:38.990Z | 4: {async_block#0}
7752 | 2025-08-25T23:14:38.990Z | at /work/oxidecomputer/omicron/clients/sled-agent-client/src/lib.rs:369:14
7753 | 2025-08-25T23:14:38.990Z | 5: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>
7754 | 2025-08-25T23:14:38.990Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7755 | 2025-08-25T23:14:38.990Z | 6: {async_fn#0}
7756 | 2025-08-25T23:14:38.990Z | at ./tests/integration_tests/instances.rs:7211:72
7757 | 2025-08-25T23:14:38.990Z | 7: {async_block#0}
7758 | 2025-08-25T23:14:38.990Z | at ./tests/integration_tests/instances.rs:1290:52
7759 | 2025-08-25T23:14:38.990Z | 8: {async_fn#0}
7760 | 2025-08-25T23:14:38.990Z | at ./tests/integration_tests/instances.rs:1307:71
7761 | 2025-08-25T23:14:38.990Z | 9: {async_block#0}
7762 | 2025-08-25T23:14:38.991Z | at ./tests/integration_tests/instances.rs:1237:1
7763 | 2025-08-25T23:14:38.991Z | 10: poll<&mut dyn core::future::future::Future<Output=()>>
7764 | 2025-08-25T23:14:38.991Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7765 | 2025-08-25T23:14:38.991Z | 11: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7766 | 2025-08-25T23:14:38.991Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7767 | 2025-08-25T23:14:38.991Z | 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7768 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:70
7769 | 2025-08-25T23:14:38.991Z | 13: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7770 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
7771 | 2025-08-25T23:14:38.991Z | 14: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7772 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
7773 | 2025-08-25T23:14:38.991Z | 15: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7774 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:25
7775 | 2025-08-25T23:14:38.991Z | 16: tokio::runtime::scheduler::current_thread::Context::enter
7776 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
7777 | 2025-08-25T23:14:38.991Z | 17: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7778 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:741:44
7779 | 2025-08-25T23:14:38.991Z | 18: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
7780 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
7781 | 2025-08-25T23:14:38.991Z | 19: tokio::runtime::context::scoped::Scoped<T>::set
7782 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
7783 | 2025-08-25T23:14:38.991Z | 20: tokio::runtime::context::set_scheduler::{{closure}}
7784 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
7785 | 2025-08-25T23:14:38.991Z | 21: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7786 | 2025-08-25T23:14:38.991Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
7787 | 2025-08-25T23:14:38.991Z | 22: std::thread::local::LocalKey<T>::with
7788 | 2025-08-25T23:14:38.991Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
7789 | 2025-08-25T23:14:38.991Z | 23: tokio::runtime::context::set_scheduler
7790 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
7791 | 2025-08-25T23:14:38.991Z | 24: tokio::runtime::scheduler::current_thread::CoreGuard::enter
7792 | 2025-08-25T23:14:38.991Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
7793 | 2025-08-25T23:14:38.991Z | 25: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
7794 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
7795 | 2025-08-25T23:14:38.992Z | 26: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7796 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
7797 | 2025-08-25T23:14:38.992Z | 27: tokio::runtime::context::runtime::enter_runtime
7798 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
7799 | 2025-08-25T23:14:38.992Z | 28: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7800 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
7801 | 2025-08-25T23:14:38.992Z | 29: tokio::runtime::runtime::Runtime::block_on_inner
7802 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
7803 | 2025-08-25T23:14:38.992Z | 30: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7804 | 2025-08-25T23:14:38.992Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
7805 | 2025-08-25T23:14:38.992Z | 31: test_instance_failed_when_on_expunged_sled
7806 | 2025-08-25T23:14:38.992Z | at ./tests/integration_tests/instances.rs:1237:1
7807 | 2025-08-25T23:14:38.992Z | 32: test_all::integration_tests::instances::test_instance_failed_when_on_expunged_sled::{{closure}}
7808 | 2025-08-25T23:14:38.992Z | at ./tests/integration_tests/instances.rs:1240:2
7809 | 2025-08-25T23:14:39.027Z | 33: core::ops::function::FnOnce::call_once
7810 | 2025-08-25T23:14:39.027Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7811 | 2025-08-25T23:14:39.027Z | 34: core::ops::function::FnOnce::call_once
7812 | 2025-08-25T23:14:39.027Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
7813 | 2025-08-25T23:14:39.027Z | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7814 | 2025-08-25T23:14:39.027Z |  

...

Start with integration_tests::instances::test_instance_failed_when_on_expunged_sled and the panic at sled-agent/src/sim/collection.rs:274; inspect the call path through sled-agent/src/sim/sled_agent.rs:523 and sled-agent/src/sim/http_entrypoints.rs:823. Reproduce the failure using the linked pull request 8878 CI check and log, then identify the cause and verify the test no longer fails unexpectedly.