oxidecomputer / oxidecomputer/omicron

test failed in CI: `test_cannot_provision_instance_beyond_cpu_capacity`

Open
#9,800 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/9781:

https://github.com/oxidecomputer/omicron/pull/9781/checks?check_run_id=62573235108

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KGNZM3F2H5XQVBNWR2QS2D3H/UVljnOkgd5TsOWoeNUgm9q0sDcwNaQb3SrkUq5JYreYvLj3k/01KGNZMNJ2K7B731E6CQY5KV6S

Excerpt from the log showing the failure:

failures:
	       integration_tests::instances::test_cannot_provision_instance_beyond_cpu_capacity
	
	    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 545 filtered out; finished in 359.99s
	    
	  stderr ───
	    log file: /var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.0.log
	    note: configured to log to "/var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.0.log"
	    DB URL: postgresql://root@[::1]:37376/omicron?sslmode=disable
	    DB address: [::1]:37376
	    log file: /var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.2.log
	    note: configured to log to "/var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.2.log"
	
	    thread 'integration_tests::instances::test_cannot_provision_instance_beyond_cpu_capacity' (2) panicked at nexus/tests/integration_tests/instances.rs:8697:19:
	    instance 2f437c66-9df3-4fc5-b6e2-01ad563187b4 did not transition to Stopped after 320s: timed out after 320.545455837s
	    stack backtrace:
	       0: __rustc::rust_begin_unwind
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/panicking.rs:698:5
	       1: core::panicking::panic_fmt
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/panicking.rs:75:14
	       2: {async_fn#0}
	                 at ./tests/integration_tests/instances.rs:8697:19
	       3: {async_fn#0}
	                 at ./tests/integration_tests/instances.rs:8648:6
	       4: {async_fn#0}
	                 at ./tests/integration_tests/instances.rs:6863:74
	       5: {async_block#0}
	                 at ./tests/integration_tests/instances.rs:6793:1
	       6: poll<&mut dyn core::future::future::Future<Output=()>>
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/future/future.rs:133:9
	       7: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/future/future.rs:133:9
	       8: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:753:70
	       9: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:167:5
	      10: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:133:5
	      11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:753:25
	      12: tokio::runtime::scheduler::current_thread::Context::enter
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:442:19
	      13: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:752:44
	      14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:840:68
	      15: tokio::runtime::context::scoped::Scoped<T>::set
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/scoped.rs:40:9
	      16: tokio::runtime::context::set_scheduler::{{closure}}
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:38
	      17: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/thread/local.rs:315:12
	      18: std::thread::local::LocalKey<T>::with
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/thread/local.rs:279:20
	      19: tokio::runtime::context::set_scheduler
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:17
	      20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:840:27
	      21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:740:24
	      22: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:200:33
	      23: tokio::runtime::context::runtime::enter_runtime
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/runtime.rs:65:16
	      24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/current_thread/mod.rs:188:9
	      25: tokio::runtime::runtime::Runtime::block_on_inner
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/runtime.rs:368:52
	      26: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
	                 at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/runtime.rs:342:18
	      27: test_cannot_provision_instance_beyond_cpu_capacity
	                 at ./tests/integration_tests/instances.rs:6793:1
	      28: test_all::integration_tests::instances::test_cannot_provision_instance_beyond_cpu_capacity::{{closure}}
	                 at ./tests/integration_tests/instances.rs:6796:2
	      29: core::ops::function::FnOnce::call_once
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/ops/function.rs:250:5
	      30: core::ops::function::FnOnce::call_once
	                 at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/ops/function.rs:250:5
	    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
	    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
	    WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpQsGFFF"
	    	If you would like to access the database for debugging, run the following:
	
	    	# Run the database
	    	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpQsGFFF/data"
	    	# Access the database. Note the port may change if you run multiple databases.
	    	cockroach sql --host=localhost:32221 --insecure
	    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 29619) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.1-clickhouse-kVe0pg)
	    failed to clean up ClickHouse data dir:
	    - /var/tmp/omicron_tmp/test_all-06a5e668aefc4dc5-test_cannot_provision_instance_beyond_cpu_capacity.29606.1-clickhouse-kVe0pg: File exists (os error 17)
	    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
	    WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpV9cvNS
	    WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
	    WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpkECUfF

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nexus/tests/integration_tests/instances.rs around the test definition at lines 6793-6796 and the timeout at line 8697. Review the linked CI log and reproduce test_cannot_provision_instance_beyond_cpu_capacity. Done means the test completes without timing out while checking the instance reaches Stopped.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.