oxidecomputer / oxidecomputer/omicron

test failed in CI: test_disk_move_between_instances

Open
#9,203 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on #9197:
https://github.com/oxidecomputer/omicron/pull/9197/checks?check_run_id=52494170053

Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01K7877FD921YHG7MRNBWWFJ9F/IwF5mrQGUnDV623PGprZasBFHEUUt6nIuqNJxN3bEHyP7idk/01K7877SEBB5ZT52166EKXPQY9#S7698

Excerpt from the log showing the failure:

        FAIL [ 360.193s] omicron-nexus::test_all integration_tests::disks::test_disk_move_between_instances
  stdout ───
    running 1 test
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: System }, version: Some(RouterVersion { router_id: a8a83e2b-1598-4353-b0b5-4bdfbcf0cc3a, version: 3 }), routes: {ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: System }, version: Some(RouterVersion { router_id: a8a83e2b-1598-4353-b0b5-4bdfbcf0cc3a, version: 4 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 })) }} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    test integration_tests::disks::test_disk_move_between_instances has been running for over 60 seconds
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
    test integration_tests::disks::test_disk_move_between_instances ... FAILED
    failures:
    failures:
        integration_tests::disks::test_disk_move_between_instances
    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 469 filtered out; finished in 359.77s
    
  stderr ───
    log file: /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.0.log
    note: configured to log to "/var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.0.log"
    DB URL: postgresql://root@[::1]:44148/omicron?sslmode=disable
    DB address: [::1]:44148
    log file: /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.2.log
    note: configured to log to "/var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.2.log"
    thread 'integration_tests::disks::test_disk_move_between_instances' panicked at nexus/tests/integration_tests/instances.rs:7697:19:
    instance 9e982e19-f25f-43cb-81f7-17501cc70b91 did not transition to Stopped after 320s: timed out after 321.172888216s
    stack backtrace:
       0: __rustc::rust_begin_unwind
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
       1: core::panicking::panic_fmt
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
       2: {async_fn#0}
                 at ./tests/integration_tests/instances.rs:7697:19
       3: {async_fn#0}
                 at ./tests/integration_tests/instances.rs:7648:6
       4: {async_fn#0}
                 at ./tests/integration_tests/disks.rs:546:10
       5: {async_block#0}
                 at ./tests/integration_tests/disks.rs:485:1
       6: poll<&mut dyn core::future::future::Future<Output=()>>
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
       7: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
       8: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:70
       9: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
      10: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
      11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:25
      12: tokio::runtime::scheduler::current_thread::Context::enter
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
      13: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:741:44
      14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
      15: tokio::runtime::context::scoped::Scoped<T>::set
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
      16: tokio::runtime::context::set_scheduler::{{closure}}
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
      17: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
      18: std::thread::local::LocalKey<T>::with
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
      19: tokio::runtime::context::set_scheduler
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
      20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
      21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
      22: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
      23: tokio::runtime::context::runtime::enter_runtime
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
      24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
      25: tokio::runtime::runtime::Runtime::block_on_inner
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
      26: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
      27: test_disk_move_between_instances
                 at ./tests/integration_tests/disks.rs:485:1
      28: test_all::integration_tests::disks::test_disk_move_between_instances::{{closure}}
                 at ./tests/integration_tests/disks.rs:486:79
      29: core::ops::function::FnOnce::call_once
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
      30: core::ops::function::FnOnce::call_once
                 at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpwCWt7k"
    	If you would like to access the database for debugging, run the following:
    	# Run the database
    	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpwCWt7k/data"
    	# Access the database. Note the port may change if you run multiple databases.
    	cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 23939) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.1-clickhouse-ub6Rg4)
    failed to clean up ClickHouse data dir:
    - /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.1-clickhouse-ub6Rg4: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpRPHu72
    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpoefmyY

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nexus/tests/integration_tests/disks.rs around the test_disk_move_between_instances call and nexus/tests/integration_tests/instances.rs:7697, then run the named integration test and inspect its log output. Trace why the instance does not reach Stopped within the timeout; done means the test completes reliably without the 320-second timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.