oxidecomputer / oxidecomputer/omicron
test failed in CI: test_cannot_bulk_write_stop_attached_disk
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on #8302:
https://github.com/oxidecomputer/omicron/runs/43824349483
Log showing the specific test failure:
Excerpt from the log showing the failure:
7963 2025-06-10T16:41:59.287Z stdout ───
7964 2025-06-10T16:41:59.287Z
7965 2025-06-10T16:41:59.287Z running 1 test
7966 2025-06-10T16:41:59.287Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: System }, version: Some(RouterVersion { router_id: a2e6c6f0-9613-49ea-b084-2927633e1483, version: 3 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(784fe521-f33a-48c0-a375-b98e908720cd)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd64:12bb:ed0b::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd64:12bb:ed0b::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(784fe521-f33a-48c0-a375-b98e908720cd)) }} }
7967 2025-06-10T16:41:59.287Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7968 2025-06-10T16:41:59.287Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7969 2025-06-10T16:41:59.288Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7970 2025-06-10T16:41:59.288Z test integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk has been running for over 60 seconds
7971 2025-06-10T16:41:59.288Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7972 2025-06-10T16:41:59.288Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7973 2025-06-10T16:41:59.288Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7974 2025-06-10T16:41:59.288Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(3207679), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
7975 2025-06-10T16:41:59.288Z test integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk ... FAILED
7976 2025-06-10T16:41:59.288Z
7977 2025-06-10T16:41:59.288Z failures:
7978 2025-06-10T16:41:59.288Z
7979 2025-06-10T16:41:59.288Z failures:
7980 2025-06-10T16:41:59.288Z integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk
7981 2025-06-10T16:41:59.288Z
7982 2025-06-10T16:41:59.288Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 443 filtered out; finished in 157.00s
7983 2025-06-10T16:41:59.288Z
7984 2025-06-10T16:41:59.288Z stderr ───
7985 2025-06-10T16:41:59.288Z log file: /var/tmp/omicron_tmp/test_all-613b6447e9724d35-test_cannot_bulk_write_stop_attached_disk.184601.0.log
7986 2025-06-10T16:41:59.288Z note: configured to log to "/var/tmp/omicron_tmp/test_all-613b6447e9724d35-test_cannot_bulk_write_stop_attached_disk.184601.0.log"
7987 2025-06-10T16:41:59.288Z DB URL: postgresql://root@[::1]:37821/omicron?sslmode=disable
7988 2025-06-10T16:41:59.288Z DB address: [::1]:37821
7989 2025-06-10T16:41:59.288Z log file: /var/tmp/omicron_tmp/test_all-613b6447e9724d35-test_cannot_bulk_write_stop_attached_disk.184601.2.log
7990 2025-06-10T16:41:59.288Z note: configured to log to "/var/tmp/omicron_tmp/test_all-613b6447e9724d35-test_cannot_bulk_write_stop_attached_disk.184601.2.log"
7991 2025-06-10T16:41:59.288Z
7992 2025-06-10T16:41:59.288Z thread 'integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk' panicked at nexus/tests/integration_tests/instances.rs:6913:19:
7993 2025-06-10T16:41:59.288Z instance b8fbb841-7e7f-4124-8cda-64abe23f6bec did not transition to Stopped after 120s: timed out after 120.642396569s
7994 2025-06-10T16:41:59.289Z stack backtrace:
7995 2025-06-10T16:41:59.289Z 0: __rustc::rust_begin_unwind
7996 2025-06-10T16:41:59.289Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
7997 2025-06-10T16:41:59.289Z 1: core::panicking::panic_fmt
7998 2025-06-10T16:41:59.289Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
7999 2025-06-10T16:41:59.289Z 2: {async_fn#0}
8000 2025-06-10T16:41:59.289Z at ./tests/integration_tests/instances.rs:6913:19
8001 2025-06-10T16:41:59.289Z 3: {async_fn#0}
8002 2025-06-10T16:41:59.289Z at ./tests/integration_tests/instances.rs:6864:6
8003 2025-06-10T16:41:59.289Z 4: {async_fn#0}
8004 2025-06-10T16:41:59.289Z at ./tests/integration_tests/pantry.rs:162:75
8005 2025-06-10T16:41:59.289Z 5: {async_fn#0}
8006 2025-06-10T16:41:59.289Z at ./tests/integration_tests/pantry.rs:816:74
8007 2025-06-10T16:41:59.289Z 6: {async_block#0}
8008 2025-06-10T16:41:59.289Z at ./tests/integration_tests/pantry.rs:794:1
8009 2025-06-10T16:41:59.289Z 7: poll<&mut dyn core::future::future::Future<Output=()>>
8010 2025-06-10T16:41:59.289Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
8011 2025-06-10T16:41:59.289Z 8: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8012 2025-06-10T16:41:59.290Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
8013 2025-06-10T16:41:59.290Z 9: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8014 2025-06-10T16:41:59.296Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:733:54
8015 2025-06-10T16:41:59.296Z 10: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8016 2025-06-10T16:41:59.296Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/task/coop/mod.rs:167:5
8017 2025-06-10T16:41:59.296Z 11: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8018 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/task/coop/mod.rs:133:5
8019 2025-06-10T16:41:59.297Z 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8020 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:733:25
8021 2025-06-10T16:41:59.297Z 13: tokio::runtime::scheduler::current_thread::Context::enter
8022 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:432:19
8023 2025-06-10T16:41:59.297Z 14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8024 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:732:36
8025 2025-06-10T16:41:59.297Z 15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
8026 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:820:68
8027 2025-06-10T16:41:59.297Z 16: tokio::runtime::context::scoped::Scoped<T>::set
8028 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/context/scoped.rs:40:9
8029 2025-06-10T16:41:59.297Z 17: tokio::runtime::context::set_scheduler::{{closure}}
8030 2025-06-10T16:41:59.297Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/context.rs:176:26
8031 2025-06-10T16:41:59.297Z 18: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
8032 2025-06-10T16:41:59.298Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:311:12
8033 2025-06-10T16:41:59.298Z 19: std::thread::local::LocalKey<T>::with
8034 2025-06-10T16:41:59.299Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:275:15
8035 2025-06-10T16:41:59.299Z 20: tokio::runtime::context::set_scheduler
8036 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/context.rs:176:9
8037 2025-06-10T16:41:59.299Z 21: tokio::runtime::scheduler::current_thread::CoreGuard::enter
8038 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:820:27
8039 2025-06-10T16:41:59.299Z 22: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
8040 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:720:19
8041 2025-06-10T16:41:59.299Z 23: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8042 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:200:28
8043 2025-06-10T16:41:59.299Z 24: tokio::runtime::context::runtime::enter_runtime
8044 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/context/runtime.rs:65:16
8045 2025-06-10T16:41:59.299Z 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8046 2025-06-10T16:41:59.299Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/scheduler/current_thread/mod.rs:188:9
8047 2025-06-10T16:41:59.299Z 26: tokio::runtime::runtime::Runtime::block_on_inner
8048 2025-06-10T16:41:59.300Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/runtime.rs:356:47
8049 2025-06-10T16:41:59.300Z 27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8050 2025-06-10T16:41:59.300Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.0/src/runtime/runtime.rs:330:13
8051 2025-06-10T16:41:59.300Z 28: test_cannot_bulk_write_stop_attached_disk
8052 2025-06-10T16:41:59.300Z at ./tests/integration_tests/pantry.rs:794:1
8053 2025-06-10T16:41:59.301Z 29: test_all::integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk::{{closure}}
8054 2025-06-10T16:41:59.301Z at ./tests/integration_tests/pantry.rs:797:2
8055 2025-06-10T16:41:59.301Z 30: core::ops::function::FnOnce::call_once
8056 2025-06-10T16:41:59.301Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
8057 2025-06-10T16:41:59.301Z 31: core::ops::function::FnOnce::call_once
8058 2025-06-10T16:41:59.301Z at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the failing integration_tests::pantry::test_cannot_bulk_write_stop_attached_disk in nexus/tests/integration_tests/pantry.rs and the timeout at nexus/tests/integration_tests/instances.rs:6913. Reproduce the test and inspect the linked CI and log output to determine why the instance did not reach Stopped within 120 seconds. Done means the failure is understood and the test passes reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, postgresql, rust
- Domain
- backend, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100