oxidecomputer / oxidecomputer/omicron

test_db_run flake

Open
#2,624 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Seen on a PR unrelated to the DB:

https://buildomat.eng.oxide.computer/wg/0/details/01GW04J1FE3KA9NX0224S52DC7/rMxcBoMeSisDyBmDYBp3wqgJmjO0yKUGFXlA56ALZYQagYmR/01GW04KEVP2Y4ENXDFGZ82HVGN

5077	2023-03-20T19:36:53.888Z	running 6 tests
5078	2023-03-20T19:36:53.898Z	test test_omicron_dev_bad_cmd ... ok
5079	2023-03-20T19:36:53.913Z	test test_omicron_dev_db_populate_no_args ... ok
5080	2023-03-20T19:36:53.956Z	test test_omicron_dev_db_wipe_no_args ... ok
5081	2023-03-20T19:36:53.985Z	test test_omicron_dev_no_args ... ok
5082	2023-03-20T19:37:51.047Z	test test_db_killed has been running for over 60 seconds
5083	2023-03-20T19:37:51.055Z	test test_db_run has been running for over 60 seconds
5084	2023-03-20T19:37:58.414Z	test test_db_killed ... ok
5085	2023-03-20T19:39:14.692Z	test test_db_run ... FAILED
5086	2023-03-20T19:39:14.698Z	
5087	2023-03-20T19:39:14.704Z	failures:
5088	2023-03-20T19:39:14.709Z	
5089	2023-03-20T19:39:14.715Z	---- test_db_run stdout ----
5090	2023-03-20T19:39:14.720Z	will run: bash -c '( set -o monitor; /work/oxidecomputer/omicron/target/debug/omicron-dev db-run --listen-port 0)'
5091	2023-03-20T19:39:14.726Z	waiting for stdout from child process
5092	2023-03-20T19:39:14.732Z	found temporary directory: /var/tmp/omicron_tmp/.tmpGx0lVw
5093	2023-03-20T19:39:14.738Z	found database pid: 95530
5094	2023-03-20T19:39:14.743Z	found postgres listen URL: postgresql://root@127.0.0.1:44891/omicron?sslmode=disable
5095	2023-03-20T19:39:14.749Z	found database populated
5096	2023-03-20T19:39:14.756Z	running db-populate
5097	2023-03-20T19:39:14.761Z	exit status: Exited(1)
5098	2023-03-20T19:39:14.768Z	stdout: "omicron-dev: populating database\n"
5099	2023-03-20T19:39:14.773Z	stdout: "omicron-dev: populating Omicron database: db error: ERROR: database \"omicron\" already exists: ERROR: database \"omicron\" already exists\n"
5100	2023-03-20T19:39:14.780Z	running db-populate --wipe
5101	2023-03-20T19:39:14.787Z	running db-wipe
5102	2023-03-20T19:39:14.803Z	cleaned up connection
5103	2023-03-20T19:39:14.811Z	sending SIGINT to process group 95451
5104	2023-03-20T19:39:14.817Z	thread 'test_db_run' panicked at 'timed out waiting 15s for process to exit', test-utils/tests/test_omicron_dev.rs:138:13
5105	2023-03-20T19:39:14.823Z	note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
5106	2023-03-20T19:39:14.828Z	
5107	2023-03-20T19:39:14.833Z	
5108	2023-03-20T19:39:14.839Z	failures:
5109	2023-03-20T19:39:14.844Z	    test_db_run
5110	2023-03-20T19:39:14.850Z	
5111	2023-03-20T19:39:14.859Z	test result: FAILED. 5 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 143.64s
5112	2023-03-20T19:39:14.867Z	
5113	2023-03-20T19:39:14.874Z	error: test failed, to rerun pass `-p omicron-test-utils --test test_omicron_dev`
5114	2023-03-20T19:39:14.882Z	     Running `/work/oxidecomputer/omicron/target/debug/deps/oxide_client-0df883f1449dbd39`

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in test-utils/tests/test_omicron_dev.rs, especially the test_db_run failure around line 138, and reproduce it with the omicron-test-utils test_omicron_dev test. Trace the omicron-dev db-run and db-populate steps shown in the log. Done means test_db_run completes reliably, including cleanup after the database-population failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.