oxidecomputer / oxidecomputer/omicron
many VMs on dogfood in state "stopped" with intended state "running"
Open
@hawkw is already working on this.
Since Oct 8, 2025.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Before testing self-service update on dogfood, we went to look at the state of running instances. We found a number of instances in state "stopped" and intended state "running":
root@oxz_switch0:~# omdb db instances | awk '$2 != $3'
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (196.0.0)
ID STATE INTENT PROPOLIS_ID SLED_ID HOST_SERIAL NAME
016aa266-b6ce-4281-ad95-245567f32f99 stopped running <no active Propolis> <not on any sled> - stress-64
055cc99b-88b8-426c-989f-60922ab4d347 stopped running <no active Propolis> <not on any sled> - deep41
0d84cbef-6197-453f-8111-278e60ffc4e3 stopped running <no active Propolis> <not on any sled> - illumos-worker
15192586-4c06-4659-a875-e9aa07ff0f03 stopped running <no active Propolis> <not on any sled> - netbooty
24ec5deb-aae5-41d0-9e9e-06e0cbca8054 stopped running <no active Propolis> <not on any sled> - stress-11
264a1e4c-4c62-45fe-992e-c45159672920 stopped guest_shutdown <no active Propolis> <not on any sled> - large-ixi
30a28d95-10bd-4729-96c5-073f46942701 stopped guest_shutdown <no active Propolis> <not on any sled> - win2022-test
3aa28350-e135-48d9-a75f-24700e5310ca stopped running <no active Propolis> <not on any sled> - x2-sdk-vm
3b6e3d0a-8714-44a8-aff3-03bf66915241 stopped running <no active Propolis> <not on any sled> - greybox
3d93d4c6-0553-4d13-be02-8e58e9a2ddef stopped running <no active Propolis> <not on any sled> - stress-91
41f7abc5-4f6c-4241-a265-290dcec8ce63 stopped running <no active Propolis> <not on any sled> - stress-87
42015600-2bcc-467a-879e-1a22ab7be4ed stopped running <no active Propolis> <not on any sled> - x2
43a4b078-a233-4c92-9438-e3e5a10d64cc stopped running <no active Propolis> <not on any sled> - stress-19
43cbab4e-dd0d-48d0-8e9d-db2ce7c635ee stopped running <no active Propolis> <not on any sled> - stress-49
43ce0389-de68-459a-9f7e-370a04ccb562 stopped running <no active Propolis> <not on any sled> - stress-76
46c99e69-15ea-45e2-8867-b06f1352adc2 stopped guest_shutdown <no active Propolis> <not on any sled> - rocky-9
59a2bf1f-5a4f-4ee0-8b5d-f35e3471f883 stopped running <no active Propolis> <not on any sled> - stress-62
5d30e534-475b-4400-9dce-69ef506fddad stopped running <no active Propolis> <not on any sled> - analytics1-worker-6
620e3d04-63a8-4f6f-b3a8-65be81008324 stopped running <no active Propolis> <not on any sled> - stress-2
6a21c8ad-fdfa-4af0-8b7a-625ae972764c stopped running <no active Propolis> <not on any sled> - stress-46
6f2c91ba-1de0-4026-9151-60210c5c9a94 stopped running <no active Propolis> <not on any sled> - stress-74
7050b318-caa2-4c96-a1d7-81cb5563d924 stopped running <no active Propolis> <not on any sled> - stress-79
75888c3c-aa40-483c-b9a0-96b37ac7c999 stopped running <no active Propolis> <not on any sled> - instance
8e9eb8e2-b9c6-4634-a0bb-b45a010a2708 stopped running <no active Propolis> <not on any sled> - htest
98ec9269-b7a0-48c1-aa86-00974f2ba1f5 stopped running <no active Propolis> <not on any sled> - stress-17
9d2135fe-d35f-4b11-bc01-41a5fa739980 stopped running <no active Propolis> <not on any sled> - stress-66
a38e0cc5-4b20-4ed1-8549-7ab5b747a2fc stopped running <no active Propolis> <not on any sled> - asfdlksdkjfasdlfkj
a43d9190-ec16-4508-86fd-24911abb3661 stopped running <no active Propolis> <not on any sled> - stress-41
b00ffc86-1e57-4f12-8f33-6c93fb27ddc9 stopped running <no active Propolis> <not on any sled> - stress-85
b4dadad1-7104-4cb7-adf0-dd204c2ba6b2 stopped running <no active Propolis> <not on any sled> - stress-35
cec9ec56-c661-4321-aa60-5c75e77591bb stopped guest_shutdown <no active Propolis> <not on any sled> - large-ixi-2
d1084954-cf74-4793-8788-89024122e50c stopped running <no active Propolis> <not on any sled> - post-malone-implies-pre-malone
e868380f-17d5-47d8-93cf-d3f1801b6b93 stopped running <no active Propolis> <not on any sled> - stress-82
f0174fbc-a8e7-4629-b950-721c60d71bbd stopped running <no active Propolis> <not on any sled> - analytics1-worker-0
f421377b-4b29-4870-b816-fb96627a6849 stopped running <no active Propolis> <not on any sled> - stress-34
fd3dcabf-c7c1-4a7a-979d-ffd7e8f2bf99 stopped running <no active Propolis> <not on any sled> - test
All the ones that we spot-checked had their most recent VMM in saga_unwound, so this presumably relates to #9174 and #9175.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.