oxidecomputer / oxidecomputer/omicron

test failed in CI: test_join_by_ip_existing_group

Open
#9,711 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on #9709:

https://github.com/oxidecomputer/omicron/pull/9709/checks?check_run_id=61188463967

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KFKH3348YSGGXYXW3CT2SC3X/Y8x04Wz6YGyRqkpmt7jA2M7uHdkUqOSPnvbggUV3S4ujQX5Q/01KFKH4NG9MA22RVH8KR378XYN

Excerpt from the log showing the failure:

8885	2026-01-22T20:07:12.665Z	  stdout ───
8886	2026-01-22T20:07:12.665Z	
8887	2026-01-22T20:07:12.665Z	    running 1 test
8888	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8889	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: System }, version: Some(RouterVersion { router_id: 91d74552-6045-45ec-8999-46245913e5d8, version: 5 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 })) }} }
8890	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8891	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: System }, version: Some(RouterVersion { router_id: 91d74552-6045-45ec-8999-46245913e5d8, version: 6 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }} }
8892	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8893	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8894	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8895	2026-01-22T20:07:12.665Z	    test integration_tests::multicast::api::test_join_by_ip_existing_group has been running for over 60 seconds
8896	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8897	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8898	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8899	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8900	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8901	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8902	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8903	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8904	2026-01-22T20:07:12.665Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8905	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8906	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8907	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8908	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8909	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8910	2026-01-22T20:07:12.666Z	    sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8911	2026-01-22T20:07:12.666Z	    test integration_tests::multicast::api::test_join_by_ip_existing_group ... FAILED
8912	2026-01-22T20:07:12.666Z	
8913	2026-01-22T20:07:12.666Z	    failures:
8914	2026-01-22T20:07:12.666Z	
8915	2026-01-22T20:07:12.666Z	    failures:
8916	2026-01-22T20:07:12.666Z	        integration_tests::multicast::api::test_join_by_ip_existing_group
8917	2026-01-22T20:07:12.666Z	
8918	2026-01-22T20:07:12.666Z	    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 618 filtered out; finished in 357.21s
8919	2026-01-22T20:07:12.666Z	    
8920	2026-01-22T20:07:12.666Z	  stderr ───
8921	2026-01-22T20:07:12.666Z	    log file: /var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.0.log
8922	2026-01-22T20:07:12.666Z	    note: configured to log to "/var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.0.log"
8923	2026-01-22T20:07:12.666Z	    DB URL: postgresql://root@[::1]:35429/omicron?sslmode=disable
8924	2026-01-22T20:07:12.666Z	    DB address: [::1]:35429
8925	2026-01-22T20:07:12.666Z	    log file: /var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.2.log
8926	2026-01-22T20:07:12.666Z	    note: configured to log to "/var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.2.log"
8927	2026-01-22T20:07:12.666Z	    Instance existing-inst-1 in Starting state - will wait for Running then stop
8928	2026-01-22T20:07:12.666Z	    Instance existing-inst-2 in Starting state - will wait for Running then stop
8929	2026-01-22T20:07:12.666Z	    Waiting for 2 instances to finish starting...
8930	2026-01-22T20:07:12.666Z	    Instance existing-inst-1 reached Running state
8931	2026-01-22T20:07:12.666Z	    Instance existing-inst-2 reached Running state
8932	2026-01-22T20:07:12.666Z	
8933	2026-01-22T20:07:12.666Z	    thread 'integration_tests::multicast::api::test_join_by_ip_existing_group' (225602) panicked at nexus/tests/integration_tests/instances.rs:8521:19:
8934	2026-01-22T20:07:12.666Z	    instance 3301b286-46c8-453e-a539-4f5eb4b8d9ec did not transition to Stopped after 320s: timed out after 320.708415726s

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nexus/tests/integration_tests/instances.rs around line 8521 and the integration_tests::multicast::api::test_join_by_ip_existing_group failure log. Determine why an instance did not transition to Stopped within 320 seconds; done when the integration test completes successfully without the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
networking, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.