oxidecomputer / oxidecomputer/omicron

test failed in CI: test_floating_ip_create_attachment

Open
#10,016 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/10012:

https://github.com/oxidecomputer/omicron/pull/10012/checks?check_run_id=66380615120

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KKAFTD6RDV3T9JDTQ1PK8Z69/FE8NMft1s1x5gd22GVXbHFYlRNAmfT8JtJpGjmSEcGU2K4U5/01KKAFV1TZZEYJZ6YE5N2W0VGJ#S8330

Excerpt from the log showing the failure:

stdout ───
	running 1 test
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: System }, version: Some(RouterVersion { router_id: fe407e14-f551-4f7b-9d2d-519158831b39, version: 3 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(2bac91dc-dce9-4653-8462-17d4631409fd)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(2bac91dc-dce9-4653-8462-17d4631409fd)) }} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	test integration_tests::external_ips::test_floating_ip_create_attachment has been running for over 60 seconds
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
	sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(7438921), kind: Custom(V6(Ipv6Net { addr: fd63:467c:6704::, width: 64 })) }, version: None, routes: {} }
	test integration_tests::external_ips::test_floating_ip_create_attachment ... FAILED
	
    failures:

    failures:
        integration_tests::external_ips::test_floating_ip_create_attachment

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 556 filtered out; finished in 362.37s
	    
stderr ───
  log file: /var/tmp/omicron_tmp/test_all-be3221e1b6456e6e-test_floating_ip_create_attachment.26270.0.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-be3221e1b6456e6e-test_floating_ip_create_attachment.26270.0.log"
  DB URL: postgresql://root@[::1]:34897/omicron?sslmode=disable
  DB address: [::1]:34897
  log file: /var/tmp/omicron_tmp/test_all-be3221e1b6456e6e-test_floating_ip_create_attachment.26270.2.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-be3221e1b6456e6e-test_floating_ip_create_attachment.26270.2.log"

  thread 'integration_tests::external_ips::test_floating_ip_create_attachment' (2) panicked at nexus/tests/integration_tests/instances.rs:8705:19:
  instance 671c14ba-daa4-4f2c-9b85-3e5e66a28b9d did not transition to Stopped after 320s: timed out after 320.261589222s
  stack backtrace:
     0: __rustc::rust_begin_unwind
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/panicking.rs:698:5
     1: core::panicking::panic_fmt
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/panicking.rs:75:14
     2: {async_fn#0}
               at ./tests/integration_tests/instances.rs:8705:19
     3: {async_fn#0}
               at ./tests/integration_tests/instances.rs:8656:6
     4: {async_fn#0}
               at ./tests/integration_tests/external_ips.rs:805:74
     5: {async_block#0}
               at ./tests/integration_tests/external_ips.rs:730:1
     6: poll<&mut dyn core::future::future::Future<Output=()>>
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/future/future.rs:133:9
     7: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/future/future.rs:133:9
     8: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:753:70
     9: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:167:5
    10: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:133:5
    11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:753:25
    12: tokio::runtime::scheduler::current_thread::Context::enter
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:442:19
    13: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:752:44
    14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:840:68
    15: tokio::runtime::context::scoped::Scoped<T>::set
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/scoped.rs:40:9
    16: tokio::runtime::context::set_scheduler::{{closure}}
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:38
    17: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/thread/local.rs:315:12
    18: std::thread::local::LocalKey<T>::with
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/thread/local.rs:279:20
    19: tokio::runtime::context::set_scheduler
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:17
    20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:840:27
    21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:740:24
    22: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:200:33
    23: tokio::runtime::context::runtime::enter_runtime
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/runtime.rs:65:16
    24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:188:9
    25: tokio::runtime::runtime::Runtime::block_on_inner
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/runtime.rs:368:52
    26: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/runtime.rs:342:18
    27: test_floating_ip_create_attachment
               at ./tests/integration_tests/external_ips.rs:730:1
    28: test_all::integration_tests::external_ips::test_floating_ip_create_attachment::{{closure}}
               at ./tests/integration_tests/external_ips.rs:733:2
    29: core::ops::function::FnOnce::call_once
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/ops/function.rs:250:5
    30: core::ops::function::FnOnce::call_once
               at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/ops/function.rs:250:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nexus/tests/integration_tests/external_ips.rs, where test_floating_ip_create_attachment is defined, and inspect the stop-transition helper in nexus/tests/integration_tests/instances.rs around line 8705. Re-run the named integration test and review the linked CI output and temporary log to identify why the instance does not reach Stopped. Done means the test completes successfully without the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.