oxidecomputer / oxidecomputer/omicron

test failed in CI: test_update_uninitialized

Open
#7,014 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/7008:

https://github.com/oxidecomputer/omicron/pull/7008/checks?check_run_id=32685500144

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JC4DDASBJ2WPJ69PBA6XCCHH/Th0R0cFXl61qEHp7FlCTnmtfHMKy8l4gcZBCNLcFIL00NUAS/01JC4DEEWSF4SF13KZM4EFJP2M

From https://buildomat.eng.oxide.computer/wg/0/details/01JC4DDASBJ2WPJ69PBA6XCCHH/Th0R0cFXl61qEHp7FlCTnmtfHMKy8l4gcZBCNLcFIL00NUAS/01JC4DEEWSF4SF13KZM4EFJP2M#S6392in particular

Excerpt from the log showing the failure:

6392	2024-11-07T23:40:17.516Z	        FAIL [  14.347s] omicron-nexus::test_all integration_tests::updates::test_update_uninitialized
6393	2024-11-07T23:40:17.516Z	
6394	2024-11-07T23:40:17.516Z	--- STDOUT:              omicron-nexus::test_all integration_tests::updates::test_update_uninitialized ---
6395	2024-11-07T23:40:17.516Z	
6396	2024-11-07T23:40:17.516Z	running 1 test
6397	2024-11-07T23:40:17.516Z	test integration_tests::updates::test_update_uninitialized ... FAILED
6398	2024-11-07T23:40:17.516Z	
6399	2024-11-07T23:40:17.517Z	failures:
6400	2024-11-07T23:40:17.517Z	
6401	2024-11-07T23:40:17.517Z	failures:
6402	2024-11-07T23:40:17.517Z	    integration_tests::updates::test_update_uninitialized
6403	2024-11-07T23:40:17.517Z	
6404	2024-11-07T23:40:17.517Z	test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 381 filtered out; finished in 14.31s
6405	2024-11-07T23:40:17.517Z	
6406	2024-11-07T23:40:17.517Z	
6407	2024-11-07T23:40:17.517Z	--- STDERR:              omicron-nexus::test_all integration_tests::updates::test_update_uninitialized ---
6408	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.0.log
6409	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.0.log"
6410	2024-11-07T23:40:17.517Z	hint: Generated a random key:
6411	2024-11-07T23:40:17.517Z	hint:
6412	2024-11-07T23:40:17.517Z	hint:   ed25519:317be6e9847327d498582c265f44f46133df4e13879b429fcf660c2e84fc5da1
6413	2024-11-07T23:40:17.517Z	hint:
6414	2024-11-07T23:40:17.517Z	hint: To modify this repository, you will need this key. Use the -k/--key
6415	2024-11-07T23:40:17.517Z	hint: command line flag or the TUFACEOUS_KEY environment variable:
6416	2024-11-07T23:40:17.517Z	hint:
6417	2024-11-07T23:40:17.517Z	hint:   export TUFACEOUS_KEY=ed25519:317be6e9847327d498582c265f44f46133df4e13879b429fcf660c2e84fc5da1
6418	2024-11-07T23:40:17.517Z	hint:
6419	2024-11-07T23:40:17.517Z	hint: To prevent this default behavior, use --no-generate-key.
6420	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.1.log
6421	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.1.log"
6422	2024-11-07T23:40:17.517Z	DB URL: postgresql://root@[::1]:45691/omicron?sslmode=disable
6423	2024-11-07T23:40:17.517Z	DB address: [::1]:45691
6424	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.3.log
6425	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.3.log"
6426	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.4.log
6427	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.4.log"
6428	2024-11-07T23:40:17.517Z	WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6429	2024-11-07T23:40:17.517Z	WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpfnKGpK"
6430	2024-11-07T23:40:17.517Z		If you would like to access the database for debugging, run the following:
6431	2024-11-07T23:40:17.517Z	
6432	2024-11-07T23:40:17.517Z		# Run the database
6433	2024-11-07T23:40:17.517Z		cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpfnKGpK/data"
6434	2024-11-07T23:40:17.517Z		# Access the database. Note the port may change if you run multiple databases.
6435	2024-11-07T23:40:17.517Z		cockroach sql --host=localhost:32221 --insecure
6436	2024-11-07T23:40:17.517Z	WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 196466) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.2-clickhouse-QgILTI)
6437	2024-11-07T23:40:17.518Z	WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6438	2024-11-07T23:40:17.518Z	WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpYfWX7i
6439	2024-11-07T23:40:17.518Z	WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6440	2024-11-07T23:40:17.518Z	WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmp3JYePq
6441	2024-11-07T23:40:17.518Z	WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6442	2024-11-07T23:40:17.518Z	WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpFGtk5l
6443	2024-11-07T23:40:17.518Z	WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6444	2024-11-07T23:40:17.518Z	WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpJ3M5SK
6445	2024-11-07T23:40:17.518Z	Error: repository fetch should have failed with 500 error
6446	2024-11-07T23:40:17.518Z	
6447	2024-11-07T23:40:17.518Z	Caused by:
6448	2024-11-07T23:40:17.518Z	    0: making request to server
6449	2024-11-07T23:40:17.518Z	    1: client error (SendRequest)
6450	2024-11-07T23:40:17.518Z	    2: connection closed before message completed
6451	2024-11-07T23:40:17.518Z	
6452	2024-11-07T23:40:17.518Z	Stack backtrace:
6453	2024-11-07T23:40:17.518Z	   0: <E as anyhow::context::ext::StdError>::ext_context
6454	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/backtrace.rs:27:14
6455	2024-11-07T23:40:17.518Z	   1: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::context
6456	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/context.rs:54:31
6457	2024-11-07T23:40:17.518Z	   2: {async_fn#0}
6458	2024-11-07T23:40:17.518Z	             at ./test-utils/src/http_testing.rs:323:28
6459	2024-11-07T23:40:17.518Z	   3: {async_fn#0}
6460	2024-11-07T23:40:17.518Z	             at ./test-utils/src/http_testing.rs:611:40
6461	2024-11-07T23:40:17.518Z	   4: {async_block#0}
6462	2024-11-07T23:40:17.518Z	             at ./tests/integration_tests/updates.rs:84:10
6463	2024-11-07T23:40:17.518Z	   5: poll<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>
6464	2024-11-07T23:40:17.518Z	             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
6465	2024-11-07T23:40:17.518Z	   6: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6466	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
6467	2024-11-07T23:40:17.518Z	   7: with_budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
6468	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
6469	2024-11-07T23:40:17.518Z	   8: budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
6470	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
6471	2024-11-07T23:40:17.518Z	   9: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6472	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
6473	2024-11-07T23:40:17.518Z	  10: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
6474	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
6475	2024-11-07T23:40:17.519Z	  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
6476	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
6477	2024-11-07T23:40:17.519Z	  12: tokio::runtime::context::runtime::enter_runtime
6478	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
6479	2024-11-07T23:40:17.519Z	  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
6480	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
6481	2024-11-07T23:40:17.519Z	  14: tokio::runtime::runtime::Runtime::block_on_inner
6482	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:363:45
6483	2024-11-07T23:40:17.519Z	  15: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6484	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:335:13
6485	2024-11-07T23:40:17.519Z	  16: test_update_uninitialized
6486	2024-11-07T23:40:17.519Z	             at ./tests/integration_tests/updates.rs:91:5
6487	2024-11-07T23:40:17.519Z	  17: test_all::integration_tests::updates::test_update_uninitialized::{{closure}}
6488	2024-11-07T23:40:17.519Z	             at ./tests/integration_tests/updates.rs:35:41

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running integration_tests::updates::test_update_uninitialized in tests/integration_tests/updates.rs, then inspect the failure around lines 84 and 91. Read the request handling in test-utils/src/http_testing.rs at lines 323 and 611 and compare the logged connection-closed error with the expected 500 response. Done means the test reliably reaches and validates the intended 500 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.