oxidecomputer / oxidecomputer/omicron
test failed in CI: `omicron-nexus::test_all integration_tests::disks::test_disk_move_between_instances`
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on pull request #10492:
https://github.com/oxidecomputer/omicron/pull/10492/checks?check_run_id=77960279403
Log showing the specific test failure:
Excerpt from the log showing the failure:
TRY 1 FAIL [ 49.835s] (─────────) omicron-nexus::test_all integration_tests::disks::test_disk_move_between_instances
stdout ───
running 1 test
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(4504495), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(4504495), kind: System }, version: Some(RouterVersion { router_id: 217d9cea-9e51-4926-808b-7dd0f0798bb0, version: 3 }), routes: {ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd09:6c06:19ec::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd09:6c06:19ec::, width: 64 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(45f472db-3651-4a2c-9f1b-d6debad46436)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(45f472db-3651-4a2c-9f1b-d6debad46436)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(4504495), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
test integration_tests::disks::test_disk_move_between_instances ... FAILED
failures:
failures:
integration_tests::disks::test_disk_move_between_instances
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 561 filtered out; finished in 49.26s
stderr ───
log file: /var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.0.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.0.log"
DB URL: postgresql://root@[::1]:56455/omicron?sslmode=disable
DB address: [::1]:56455
log file: /var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.2.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.2.log"
thread 'integration_tests::disks::test_disk_move_between_instances' (2) panicked at sled-agent/src/sim/collection.rs:274:65:
called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:48313/vmms/4bc8315a-00b6-4021-9cb4-549cfc92a2c9)" }
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
1: core::panicking::panic_fmt
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
2: core::result::unwrap_failed
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1867:5
3: unwrap<(), omicron_common::api::external::error::Error>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1233:23
4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:65
5: {async_fn#0}
at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:571:58
6: {async_fn#0}
at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:1244:38
7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_types_versions::v1::instance::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_types_versions::v1::instance::VmmPathParam>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.0/src/handler.rs:664:78
8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_types_versions::v1::instance::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_types_versions::v1::instance::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.0/src/handler.rs:793:16
10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.0/src/server.rs:945:69
12: <tokio::runtime::task::core::Core<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/core.rs:375:24
13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/loom/std/unsafe_cell.rs:16:9
14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/core.rs:364:30
15: tokio::runtime::task::harness::poll_future::<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/harness.rs:535:30
16: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panic/unwind_safe.rs:274:9
17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:581:40
18: __rust_try
19: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:544:19
20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panic.rs:359:14
21: tokio::runtime::task::harness::poll_future::<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/harness.rs:523:18
22: <tokio::runtime::task::harness::Harness<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll_inner
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/harness.rs:210:27
23: <tokio::runtime::task::harness::Harness<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/harness.rs:155:20
24: tokio::runtime::task::raw::poll::<dropshot::server::http_request_handle<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent>>::{closure#0}::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/raw.rs:337:13
25: <tokio::runtime::task::raw::RawTask>::poll
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/raw.rs:267:18
26: <tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::run
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/task/mod.rs:515:13
27: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:825:30
28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
29: budget<(), tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:373:43
31: <tokio::runtime::scheduler::current_thread::Context>::enter::<(), <tokio::runtime::scheduler::current_thread::Context>::run_task<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>::{closure#0}>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:451:19
32: <tokio::runtime::scheduler::current_thread::Context>::run_task::<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:373:28
33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:821:43
34: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:68
35: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/scoped.rs:40:9
36: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:38
37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:513:12
38: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:477:20
39: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:17
40: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:27
41: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:765:24
42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:205:33
43: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:193:9
45: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:371:52
46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
47: test_disk_move_between_instances
at ./tests/integration_tests/disks.rs:518:1
48: test_all::integration_tests::disks::test_disk_move_between_instances::{closure#0}
at ./tests/integration_tests/disks.rs:519:79
49: <test_all::integration_tests::disks::test_disk_move_between_instances::{closure#0} as core::ops::function::FnOnce<()>>::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
50: core::ops::function::FnOnce::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'integration_tests::disks::test_disk_move_between_instances' (2) panicked at clients/sled-agent-client/src/lib.rs:256:14:
instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:53417/vmms/4bc8315a-00b6-4021-9cb4-549cfc92a2c9/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
1: core::panicking::panic_fmt
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
2: core::result::unwrap_failed
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1867:5
3: <core::result::Result<(), reqwest::error::Error>>::expect
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1185:23
4: {async_block#0}
at /work/oxidecomputer/omicron/clients/sled-agent-client/src/lib.rs:256:14
5: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
6: {async_fn#0}
at ./tests/integration_tests/disks.rs:226:62
7: {async_fn#0}
at ./tests/integration_tests/disks.rs:539:44
8: {async_block#0}
at ./tests/integration_tests/disks.rs:518:1
9: poll<&mut dyn core::future::future::Future<Output=()>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
10: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:70
12: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
13: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:25
15: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#0}>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:451:19
16: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:777:44
17: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:68
18: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/scoped.rs:40:9
19: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:38
20: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:513:12
21: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/thread/local.rs:477:20
22: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:17
23: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:27
24: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:765:24
25: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:205:33
26: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:193:9
28: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:371:52
29: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
30: test_disk_move_between_instances
at ./tests/integration_tests/disks.rs:518:1
31: test_all::integration_tests::disks::test_disk_move_between_instances::{closure#0}
at ./tests/integration_tests/disks.rs:519:79
32: <test_all::integration_tests::disks::test_disk_move_between_instances::{closure#0} as core::ops::function::FnOnce<()>>::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
33: core::ops::function::FnOnce::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpbN9QwL"
If you would like to access the database for debugging, run the following:
# Run the database
cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpbN9QwL/data"
# Access the database. Note the port may change if you run multiple databases.
cockroach sql --host=localhost:32221 --insecure
WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 18695) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.1-clickhouse-qDZwez)
failed to clean up ClickHouse data dir:
- /var/tmp/omicron_tmp/test_all-cfd0298613e2de83-test_disk_move_between_instances.18686.1-clickhouse-qDZwez: File exists (os error 17)
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmp6X9KSp
WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpbeP9uB
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with integration_tests::disks::test_disk_move_between_instances and inspect the CI log around sled-agent/src/sim/collection.rs:274, where the simulated instance request fails. Reproduce the named test and trace the communication error to determine what change is needed; done means the test passes reliably in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100