oxidecomputer / oxidecomputer/omicron

many VMs on dogfood in state "stopped" with intended state "running"

Open
#9,177 1 comment 0 reactions 1 assignee View on GitHub

@hawkw is already working on this.

Since Oct 8, 2025.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Before testing self-service update on dogfood, we went to look at the state of running instances. We found a number of instances in state "stopped" and intended state "running":

root@oxz_switch0:~# omdb db instances | awk '$2 != $3'
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (196.0.0)
ID                                   STATE   INTENT         PROPOLIS_ID                          SLED_ID                              HOST_SERIAL NAME                           
016aa266-b6ce-4281-ad95-245567f32f99 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-64                      
055cc99b-88b8-426c-989f-60922ab4d347 stopped running        <no active Propolis>                 <not on any sled>                    -           deep41                         
0d84cbef-6197-453f-8111-278e60ffc4e3 stopped running        <no active Propolis>                 <not on any sled>                    -           illumos-worker                 
15192586-4c06-4659-a875-e9aa07ff0f03 stopped running        <no active Propolis>                 <not on any sled>                    -           netbooty                       
24ec5deb-aae5-41d0-9e9e-06e0cbca8054 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-11                      
264a1e4c-4c62-45fe-992e-c45159672920 stopped guest_shutdown <no active Propolis>                 <not on any sled>                    -           large-ixi                      
30a28d95-10bd-4729-96c5-073f46942701 stopped guest_shutdown <no active Propolis>                 <not on any sled>                    -           win2022-test                   
3aa28350-e135-48d9-a75f-24700e5310ca stopped running        <no active Propolis>                 <not on any sled>                    -           x2-sdk-vm                      
3b6e3d0a-8714-44a8-aff3-03bf66915241 stopped running        <no active Propolis>                 <not on any sled>                    -           greybox                        
3d93d4c6-0553-4d13-be02-8e58e9a2ddef stopped running        <no active Propolis>                 <not on any sled>                    -           stress-91                      
41f7abc5-4f6c-4241-a265-290dcec8ce63 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-87                      
42015600-2bcc-467a-879e-1a22ab7be4ed stopped running        <no active Propolis>                 <not on any sled>                    -           x2                             
43a4b078-a233-4c92-9438-e3e5a10d64cc stopped running        <no active Propolis>                 <not on any sled>                    -           stress-19                      
43cbab4e-dd0d-48d0-8e9d-db2ce7c635ee stopped running        <no active Propolis>                 <not on any sled>                    -           stress-49                      
43ce0389-de68-459a-9f7e-370a04ccb562 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-76                      
46c99e69-15ea-45e2-8867-b06f1352adc2 stopped guest_shutdown <no active Propolis>                 <not on any sled>                    -           rocky-9                        
59a2bf1f-5a4f-4ee0-8b5d-f35e3471f883 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-62                      
5d30e534-475b-4400-9dce-69ef506fddad stopped running        <no active Propolis>                 <not on any sled>                    -           analytics1-worker-6            
620e3d04-63a8-4f6f-b3a8-65be81008324 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-2                       
6a21c8ad-fdfa-4af0-8b7a-625ae972764c stopped running        <no active Propolis>                 <not on any sled>                    -           stress-46                      
6f2c91ba-1de0-4026-9151-60210c5c9a94 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-74                      
7050b318-caa2-4c96-a1d7-81cb5563d924 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-79                      
75888c3c-aa40-483c-b9a0-96b37ac7c999 stopped running        <no active Propolis>                 <not on any sled>                    -           instance                       
8e9eb8e2-b9c6-4634-a0bb-b45a010a2708 stopped running        <no active Propolis>                 <not on any sled>                    -           htest                          
98ec9269-b7a0-48c1-aa86-00974f2ba1f5 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-17                      
9d2135fe-d35f-4b11-bc01-41a5fa739980 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-66                      
a38e0cc5-4b20-4ed1-8549-7ab5b747a2fc stopped running        <no active Propolis>                 <not on any sled>                    -           asfdlksdkjfasdlfkj             
a43d9190-ec16-4508-86fd-24911abb3661 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-41                      
b00ffc86-1e57-4f12-8f33-6c93fb27ddc9 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-85                      
b4dadad1-7104-4cb7-adf0-dd204c2ba6b2 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-35                      
cec9ec56-c661-4321-aa60-5c75e77591bb stopped guest_shutdown <no active Propolis>                 <not on any sled>                    -           large-ixi-2                    
d1084954-cf74-4793-8788-89024122e50c stopped running        <no active Propolis>                 <not on any sled>                    -           post-malone-implies-pre-malone 
e868380f-17d5-47d8-93cf-d3f1801b6b93 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-82                      
f0174fbc-a8e7-4629-b950-721c60d71bbd stopped running        <no active Propolis>                 <not on any sled>                    -           analytics1-worker-0            
f421377b-4b29-4870-b816-fb96627a6849 stopped running        <no active Propolis>                 <not on any sled>                    -           stress-34                      
fd3dcabf-c7c1-4a7a-979d-ffd7e8f2bf99 stopped running        <no active Propolis>                 <not on any sled>                    -           test                           

All the ones that we spot-checked had their most recent VMM in saga_unwound, so this presumably relates to #9174 and #9175.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.