test failed in CI: omicron-dev::test-omicron-dev test_run_all

Failed on run https://github.com/oxidecomputer/omicron/runs/71863566326

Failed here: https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9606

[9610](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9610)	2026-04-17T17:27:27.218Z	    running 1 test
[9611](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9611)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: setting up all services ...
[9612](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9612)	2026-04-17T17:27:27.219Z	    subproc stdout: log file: /var/tmp/omicron_tmp/omicron-dev-omicron-dev.14880.0.log
[9613](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9613)	2026-04-17T17:27:27.219Z	    subproc stdout: note: configured to log to "/var/tmp/omicron_tmp/omicron-dev-omicron-dev.14880.0.log"
[9614](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9614)	2026-04-17T17:27:27.219Z	    subproc stdout: DB URL: postgresql://root@[::1]:62719/omicron?sslmode=disable
[9615](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9615)	2026-04-17T17:27:27.219Z	    subproc stdout: DB address: [::1]:62719
[9616](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9616)	2026-04-17T17:27:27.219Z	    subproc stdout: log file: /var/tmp/omicron_tmp/omicron-dev-omicron-dev.14880.2.log
[9617](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9617)	2026-04-17T17:27:27.219Z	    subproc stdout: note: configured to log to "/var/tmp/omicron_tmp/omicron-dev-omicron-dev.14880.2.log"
[9618](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9618)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: Adding disks to first sled agent
[9619](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9619)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: services are running.
[9620](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9620)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: nexus external API:     127.0.0.1:56318
[9621](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9621)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: nexus internal API:     [::1]:12221
[9622](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9622)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: nexus lockstep API:     [::1]:12232
[9623](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9623)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: sled agent API:         http://[::1]:59462
[9624](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9624)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: cockroachdb pid:        14901
[9625](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9625)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: cockroachdb URL:        postgresql://root@[::1]:62719/omicron?sslmode=disable
[9626](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9626)	2026-04-17T17:27:27.219Z	    subproc stdout: omicron-dev: cockroachdb directory:  /var/tmp/omicron_tmp/.tmpjACSQj
[9627](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9627)	2026-04-17T17:27:27.219Z	    test test_run_all has been running for over 60 seconds
[9628](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9628)	2026-04-17T17:27:27.219Z	  stderr ───
[9629](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9629)	2026-04-17T17:27:27.219Z	    will run: bash -c '( set -o monitor; /work/oxidecomputer/omicron/target/debug/omicron-dev run-all --nexus-listen-port 0 && true )'
[9630](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9630)	2026-04-17T17:27:27.219Z	    waiting for stdout from child process
[9631](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9631)	2026-04-17T17:27:27.219Z	    found Nexus external API:     127.0.0.1:56318
[9632](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9632)	2026-04-17T17:27:27.219Z	    found cockroachdb pid:        14901
[9633](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9633)	2026-04-17T17:27:27.219Z	    found postgres listen URL:        postgresql://root@[::1]:62719/omicron?sslmode=disable
[9634](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9634)	2026-04-17T17:27:27.219Z	    found cockroachdb directory:  /var/tmp/omicron_tmp/.tmpjACSQj
[9635](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9635)	2026-04-17T17:27:27.219Z	    cleaned up connection
[9636](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9636)	2026-04-17T17:27:27.219Z	    sending SIGINT to process group 14880
[9637](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9637)	2026-04-17T17:27:27.219Z	
[9638](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9638)	2026-04-17T17:27:27.219Z	    thread 'test_run_all' (2) panicked at dev-tools/omicron-dev-lib/src/test_utils.rs:29:13:
[9639](https://buildomat.eng.oxide.computer/wg/0/details/01KPE5NVNC9KTPYDYHPRD04GHC/wagAhakzpgNyZE8RCXNwbBglRvqjxvw2RhKUQl6gieH9k6RW/01KPE5PGJMY19SF83DQ30H7SKB#S9639)	2026-04-17T17:27:27.219Z	    timed out waiting 30s for process to exit

The test then timed out, restarted, and finished within five seconds.

Note that the first run took "over 900 seconds" before timing out, but passed within 5 seconds on retry? This implies it's probably actually getting wedged, unless it's really becoming pathologically slow somehow. Just clarifying that this probably isn't an "adjust-a-timeout" style flake.

Start with the test_run_all failure and the timeout panic at dev-tools/omicron-dev-lib/src/test_utils.rs:29. Run the omicron-dev run-all path and inspect the linked CI logs, focusing on why the first service shutdown waits indefinitely while the retry completes quickly. Done means the test no longer wedges and the CI run completes reliably without merely increasing the timeout.