oxidecomputer / oxidecomputer/omicron

Polling sled-agent during local disk creation may re-create a deleted disk and orphan it

Open
#10,986 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

local storage
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This incident occurred on colo back on 2026-04-03, so it's not entirely clear to me that it could recur, but I think the core logic that triggered it is unchanged.

A 2T local disk allocation was created by an instance-start saga. The instance and its disk is deleted by a separate saga, which is successful. However, the instance-start saga continue to poll local_storage_dataset_ensure, which re-creates the deleted disk, before eventually failing because the instance is now missing. The local storage allocation is already marked as deleted in the DB, so the dataset remains present on the disk.

BRM44220009 # zfs list -r oxp_a5a52f47-9c9a-4519-83dc-abc56619495d | grep -e NAME -e /vol
NAME                                                                                                           USED  AVAIL     REFER  MOUNTPOINT
oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol   2.00T   140G     2.00T  -

root@[fd00:1122:3344:116::3]:32221/omicron> select * from local_storage_unencrypted_dataset_allocation where id = '1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85';
                   id                  |         time_created          |         time_deleted          | local_storage_unencrypted_dataset_id |               pool_id                |               sled_id                | dat
aset_size
---------------------------------------+-------------------------------+-------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+----
------------
  1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85 | 2026-04-03 18:10:11.577606+00 | 2026-04-03 18:10:52.056022+00 | f9a6ee4a-84db-489b-adf4-bc2df447096e | a5a52f47-9c9a-4519-83dc-abc56619495d | 4206d03e-6177-4ea5-abb4-cad472dbdd90 | 234
9347110912
(1 row)

Time: 2ms total (execution 2ms / network 0ms)

root@[fd00:1122:3344:116::3]:32221/omicron> select * from disk_type_local_storage where local_storage_unencrypted_dataset_allocation_id = '1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85';
                disk_id                | required_dataset_overhead | local_storage_dataset_allocation_id | local_storage_unencrypted_dataset_allocation_id
---------------------------------------+---------------------------+-------------------------------------+--------------------------------------------------
  0de11f7f-8586-4449-9712-6cfd4ace5899 |              150323855360 | NULL                                | 1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
(1 row)

root@oxz_switch1:~# omdb db disks info 0de11f7f-8586-4449-9712-6cfd4ace5899
HOST_SERIAL DISK_NAME         INSTANCE_NAME PROPOLIS_ZONE DISK_STATE 
-           localjsondisk-2tb -             -             destroyed  
DISK_NAME         TIME_CREATED                   TIME_DELETED                   ALLOCATION_TYPE DATASET_ID                           POOL_ID                              SLED_ID                              DATASET_SIZE  
localjsondisk-2tb 2026-04-03 18:10:11.577606 UTC 2026-04-03 18:10:52.056022 UTC unencrypted     f9a6ee4a-84db-489b-adf4-bc2df447096e a5a52f47-9c9a-4519-83dc-abc56619495d 4206d03e-6177-4ea5-abb4-cad472dbdd90 2349347110912

Here you can see the disk created at 18:10:12, then we poll at increasing intervals, the destroy occurs at 18:10:52, but the next poll at 18:11:22 creates a new copy of the volume, and we continue polling until the disk is fully initialized.

BRM44220009 # zpool history oxp_a5a52f47-9c9a-4519-83dc-abc56619495d | grep 1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:12 zfs create -o canmount=off -o mountpoint=/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85 oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:12 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:12 zfs create -V 2199023255552 -o rawvol=on -o volblocksize=131072 oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol
2026-04-03.18:10:14 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:17 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:20 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:23 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:27 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:36 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:47 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:10:52 zfs destroy oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol
2026-04-03.18:10:57 zfs destroy -r oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:11:22 zfs create -o canmount=off -o mountpoint=/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85 oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:11:22 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:11:22 zfs create -V 2199023255552 -o rawvol=on -o volblocksize=131072 oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol
2026-04-03.18:12:29 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:14:56 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:17:14 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85
2026-04-03.18:19:27 zfs set quota=2349347110912 reservation=2349347110912 compression=off oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85

I think this is occurring because we poll for disk creation by continuing to send POST requests to sled-agent. A 503 due to disk initialization on large local disks is normal, so we frequently end up polling for multiple minutes, and if the disk is deleted out from under us between poll intervals we don't notice.

2026-04-03 18:10:35.874Z INFO SledAgent/656 (dropshot (SledAgent)) on BRM44220009: request completed
    error_message_external = Service Unavailable
    error_message_internal = Failed to ensure volume 'oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol': created but not ready yet: still zero initializing
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:855
    latency_us = 1967463
    local_addr = [fd00:1122:3344:11d::1]:12345
    method = POST
    remote_addr = [fd00:1122:3344:11d::22]:39172
    req_id = 01f27d2f-f1dd-4c69-b504-cca8a4d010c0
    response_code = 503
    uri = /local-storage
2026-04-03 18:10:46.758Z INFO SledAgent/656 (dropshot (SledAgent)) on BRM44220009: request completed
    error_message_external = Service Unavailable
    error_message_internal = Failed to ensure volume 'oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol': created but not ready yet: still zero initializing
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:855
    latency_us = 1891818
    local_addr = [fd00:1122:3344:11d::1]:12345
    method = POST
    remote_addr = [fd00:1122:3344:11d::22]:39172
    req_id = 98ff17a9-a0be-43fa-a4ba-4fc6bafd8d71
    response_code = 503
    uri = /local-storage
2026-04-03 18:11:22.530Z INFO SledAgent/656 (dropshot (SledAgent)) on BRM44220009: request completed
    error_message_external = Service Unavailable
    error_message_internal = Failed to ensure volume 'oxp_a5a52f47-9c9a-4519-83dc-abc56619495d/local_storage_unencrypted/1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85/vol': created but not ready yet: still zero initializing
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:855
    latency_us = 161239
    local_addr = [fd00:1122:3344:11d::1]:12345
    method = POST
    remote_addr = [fd00:1122:3344:11d::22]:63618
    req_id = 2fb763c8-2b76-4579-8150-0f05ad4d96f2
    response_code = 503
    uri = /local-storage

The instance-start saga:

 id                                   | current_sec                          | time_created             | name           | state 
--------------------------------------+--------------------------------------+--------------------------+----------------+-------
 6618297e-9b9f-4f60-b16f-e411180eac5e | f3f20c9b-eb7b-4540-8eda-ac051b7f2c22 | 2026-04-03T18:10:11.269Z | instance-start | Done  

DAG: {"end_node":25,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[5,6,null],[5,7,null],[5,8,null],[5,9,null],[5,10,null],[5,11,null],[5,12,null],[5,13,null],[5,14,null],[
5,15,null],[5,16,null],[5,17,null],[6,18,null],[7,18,null],[8,18,null],[9,18,null],[10,18,null],[11,18,null],[12,18,null],[13,18,null],[14,18,null],[15,18,null],[16,18,null],[17,18,null],[18,19,null],[19,20,null],[20,21,null]
,[21,22,null],[22,23,null],[24,0,null],[23,25,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"instance_start.generate_propolis_id","label":"GeneratePropolisId","name":"propolis_id"}},{"Action":{"action_name":"instan
ce_start.alloc_server","label":"AllocServer","name":"sled_id"}},{"Action":{"action_name":"instance_start.alloc_propolis_ip","label":"AllocPropolisIp","name":"propolis_ip"}},{"Action":{"action_name":"instance_start.create_vmm_
record","label":"CreateVmmRecord","name":"vmm_record"}},{"Action":{"action_name":"instance_start.mark_as_starting","label":"MarkAsStarting","name":"started_record"}},{"Action":{"action_name":"instance_start.list_local_storage
","label":"ListLocalStorage","name":"local_storage_records"}},{"Action":{"action_name":"instance_start.ensure_local_storage_0","label":"EnsureLocalStorage_0","name":"ensure_local_storage_0"}},{"Action":{"action_name":"instanc
e_start.ensure_local_storage_1","label":"EnsureLocalStorage_1","name":"ensure_local_storage_1"}},{"Action":{"action_name":"instance_start.ensure_local_storage_2","label":"EnsureLocalStorage_2","name":"ensure_local_storage_2"}
},{"Action":{"action_name":"instance_start.ensure_local_storage_3","label":"EnsureLocalStorage_3","name":"ensure_local_storage_3"}},{"Action":{"action_name":"instance_start.ensure_local_storage_4","label":"EnsureLocalStorage_
4","name":"ensure_local_storage_4"}},{"Action":{"action_name":"instance_start.ensure_local_storage_5","label":"EnsureLocalStorage_5","name":"ensure_local_storage_5"}},{"Action":{"action_name":"instance_start.ensure_local_stor
age_6","label":"EnsureLocalStorage_6","name":"ensure_local_storage_6"}},{"Action":{"action_name":"instance_start.ensure_local_storage_7","label":"EnsureLocalStorage_7","name":"ensure_local_storage_7"}},{"Action":{"action_name
":"instance_start.ensure_local_storage_8","label":"EnsureLocalStorage_8","name":"ensure_local_storage_8"}},{"Action":{"action_name":"instance_start.ensure_local_storage_9","label":"EnsureLocalStorage_9","name":"ensure_local_s
torage_9"}},{"Action":{"action_name":"instance_start.ensure_local_storage_10","label":"EnsureLocalStorage_10","name":"ensure_local_storage_10"}},{"Action":{"action_name":"instance_start.ensure_local_storage_11","label":"Ensur
eLocalStorage_11","name":"ensure_local_storage_11"}},{"Action":{"action_name":"instance_start.dpd_ensure","label":"DpdEnsure","name":"dpd_ensure"}},{"Action":{"action_name":"instance_start.v2p_ensure","label":"V2PEnsure","nam
e":"v2p_ensure"}},{"Action":{"action_name":"instance_start.ensure_registered","label":"EnsureRegistered","name":"ensure_registered"}},{"Action":{"action_name":"instance_start.update_multicast_sled_id","label":"UpdateMulticast
SledId","name":"multicast_sled_id"}},{"Action":{"action_name":"instance_start.add_virtual_resources","label":"AddVirtualResources","name":"virtual_resources"}},{"Action":{"action_name":"instance_start.ensure_running","label":
"EnsureRunning","name":"ensure_running"}},{"Start":{"params":{"db_instance":{"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":"89d3e9f6-2029-4dde-8b9d-a6a94ecf4d90","cpu_platform":null,"hostname":"test-host","ide
ntity":{"description":"test-instance","id":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","name":"test-instance-for-disks","time_created":"2026-04-03T18:06:36.579290Z","time_deleted":null,"time_modified":"2026-04-03T18:07:45.158210Z"
},"intended_state":"Stopped","memory":17179869184,"ncpus":4,"project_id":"741c3913-77a7-46c7-8600-3cddc1737ccb","runtime_state":{"dst_propolis_id":null,"gen":4,"migration_id":null,"nexus_state":"NoVmm","propolis_id":null,"tim
e_last_auto_restarted":null,"time_updated":"2026-04-03T18:07:51.113317Z"},"updater_gen":4,"updater_id":null,"user_data":[]},"reason":"User","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"074fc33
d-ef0b-46e9-ae15-d45b0fffba01","silo_user_id":"ebe3b1f0-634a-4ebb-82ef-991418f4e7da"}},"credential_id":"fa46b61f-27f5-46ce-8c88-76fdc178382e","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"E
nd"]},"saga_name":"instance-start","start_node":24}

              event time | sub saga | node id                                     | event type    | data
------------------------ | -------- | ------------------------------------------- | ------------- | ---
2026-04-03T18:10:11.320Z |          |                                   24: start | started       | 
2026-04-03T18:10:11.327Z |          |                                   24: start | succeeded     | 
2026-04-03T18:10:11.331Z |          |      0: instance_start.generate_propolis_id | started       | 
2026-04-03T18:10:11.334Z |          |      0: instance_start.generate_propolis_id | succeeded     | "propolis_id" => "bda655a4-15e3-48cf-b466-70ab8f458093"
2026-04-03T18:10:11.337Z |          |              1: instance_start.alloc_server | started       | 
2026-04-03T18:10:11.659Z |          |              1: instance_start.alloc_server | succeeded     | "sled_id" => "4206d03e-6177-4ea5-abb4-cad472dbdd90"
2026-04-03T18:10:11.663Z |          |         2: instance_start.alloc_propolis_ip | started       | 
2026-04-03T18:10:11.674Z |          |         2: instance_start.alloc_propolis_ip | succeeded     | "propolis_ip" => "fd00:1122:3344:11d::1:6b2"
2026-04-03T18:10:11.681Z |          |         3: instance_start.create_vmm_record | started       | 
2026-04-03T18:10:11.746Z |          |         3: instance_start.create_vmm_record | succeeded     | "vmm_record" => {"cpu_platform":"AmdMilan","id":"bda655a4-15e3-48cf-b466-70ab8f458093","instance_id":"ac0ea3b7-4ef3-4cd1-9e59
-8995347daa59","propolis_ip":"fd00:1122:3344:11d::1:6b2/128","propolis_port":12400,"runtime":{"gen":1,"state":"Creating","time_state_updated":"2026-04-03T18:10:11.694185Z"},"sled_id":"4206d03e-6177-4ea5-abb4-cad472dbdd90","ti
me_created":"2026-04-03T18:10:11.694185Z","time_deleted":null}
2026-04-03T18:10:11.752Z |          |          4: instance_start.mark_as_starting | started       | 
2026-04-03T18:10:11.870Z |          |          4: instance_start.mark_as_starting | succeeded     | "started_record" => {"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":"89d3e9f6-2029-4dde-8b9d-a6a94ecf4d90","cp
u_platform":null,"hostname":"test-host","identity":{"description":"test-instance","id":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","name":"test-instance-for-disks","time_created":"2026-04-03T18:06:36.579290Z","time_deleted":null,"
time_modified":"2026-04-03T18:10:11.212469Z"},"intended_state":"Running","memory":17179869184,"ncpus":4,"project_id":"741c3913-77a7-46c7-8600-3cddc1737ccb","runtime_state":{"dst_propolis_id":null,"gen":5,"migration_id":null,"
nexus_state":"Vmm","propolis_id":"bda655a4-15e3-48cf-b466-70ab8f458093","time_last_auto_restarted":null,"time_updated":"2026-04-03T18:07:51.113317Z"},"updater_gen":4,"updater_id":null,"user_data":[]}
2026-04-03T18:10:11.873Z |          |        5: instance_start.list_local_storage | started       | 
2026-04-03T18:10:11.940Z |          |        5: instance_start.list_local_storage | succeeded     | "local_storage_records" => [{"disk":{"block_size":"AdvancedFormat","disk_type":"LocalStorage","identity":{"description":"loca
ldisk-testing-2tb","id":"0de11f7f-8586-4449-9712-6cfd4ace5899","name":"localjsondisk-2tb","time_created":"2026-04-03T18:10:04.388696Z","time_deleted":null,"time_modified":"2026-04-03T18:10:04.388696Z"},"project_id":"741c3913-
77a7-46c7-8600-3cddc1737ccb","rcgen":1,"runtime_state":{"attach_instance_id":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","disk_state":"attached","gen":2,"time_updated":"2026-04-03T18:10:04.535353Z"},"size":2199023255552,"slot":4},
"disk_type_local_storage":{"disk_id":"0de11f7f-8586-4449-9712-6cfd4ace5899","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85","required_dataset
_overhead":150323855360},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":2349347110912,"id":"1d843bb7-4ec7-4ff4-adb7-f82d39f9fe85","local_storage_unencrypted_dataset_id":"f9a6ee4a-84db-489b-adf4-bc2df447096e
","pool_id":"a5a52f47-9c9a-4519-83dc-abc56619495d","sled_id":"4206d03e-6177-4ea5-abb4-cad472dbdd90","time_created":"2026-04-03T18:10:11.577606Z","time_deleted":null}}},{"disk":{"block_size":"AdvancedFormat","disk_type":"Local
Storage","identity":{"description":"localdisk-testing","id":"095892cb-d72b-4b89-9fa8-b1f065b0898e","name":"localjsondisk-4gb","time_created":"2026-04-03T18:10:03.062871Z","time_deleted":null,"time_modified":"2026-04-03T18:10:
03.062871Z"},"project_id":"741c3913-77a7-46c7-8600-3cddc1737ccb","rcgen":1,"runtime_state":{"attach_instance_id":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","disk_state":"attached","gen":2,"time_updated":"2026-04-03T18:10:03.35376
0Z"},"size":4294967296,"slot":3},"disk_type_local_storage":{"disk_id":"095892cb-d72b-4b89-9fa8-b1f065b0898e","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"0aeeb692-bb55-4cee-b23
3-52a6bb1041cf","required_dataset_overhead":293601280},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":4588568576,"id":"0aeeb692-bb55-4cee-b233-52a6bb1041cf","local_storage_unencrypted_dataset_id":"36e294ae-
ecb3-4866-a594-465cc453ba85","pool_id":"cbcad26e-5e52-41b7-9875-1a84d30d8a15","sled_id":"4206d03e-6177-4ea5-abb4-cad472dbdd90","time_created":"2026-04-03T18:10:11.577606Z","time_deleted":null}}}]
2026-04-03T18:10:11.944Z |          |    6: instance_start.ensure_local_storage_0 | started       | 
2026-04-03T18:10:11.948Z |          |  17: instance_start.ensure_local_storage_11 | started       | 
2026-04-03T18:10:11.951Z |          |  16: instance_start.ensure_local_storage_10 | started       | 
2026-04-03T18:10:11.954Z |          |  17: instance_start.ensure_local_storage_11 | succeeded     | 
2026-04-03T18:10:11.957Z |          |  16: instance_start.ensure_local_storage_10 | succeeded     | 
2026-04-03T18:10:11.960Z |          |   15: instance_start.ensure_local_storage_9 | started       | 
2026-04-03T18:10:11.963Z |          |   14: instance_start.ensure_local_storage_8 | started       | 
2026-04-03T18:10:11.966Z |          |   15: instance_start.ensure_local_storage_9 | succeeded     | 
2026-04-03T18:10:11.969Z |          |   14: instance_start.ensure_local_storage_8 | succeeded     | 
2026-04-03T18:10:11.972Z |          |   13: instance_start.ensure_local_storage_7 | started       | 
2026-04-03T18:10:11.975Z |          |   11: instance_start.ensure_local_storage_5 | started       | 
2026-04-03T18:10:11.978Z |          |   13: instance_start.ensure_local_storage_7 | succeeded     | 
2026-04-03T18:10:11.981Z |          |   11: instance_start.ensure_local_storage_5 | succeeded     | 
2026-04-03T18:10:11.985Z |          |   12: instance_start.ensure_local_storage_6 | started       | 
2026-04-03T18:10:11.988Z |          |   10: instance_start.ensure_local_storage_4 | started       | 
2026-04-03T18:10:11.990Z |          |   12: instance_start.ensure_local_storage_6 | succeeded     | 
2026-04-03T18:10:11.993Z |          |   10: instance_start.ensure_local_storage_4 | succeeded     | 
2026-04-03T18:10:11.997Z |          |    9: instance_start.ensure_local_storage_3 | started       | 
2026-04-03T18:10:11.999Z |          |    8: instance_start.ensure_local_storage_2 | started       | 
2026-04-03T18:10:12.002Z |          |    9: instance_start.ensure_local_storage_3 | succeeded     | 
2026-04-03T18:10:12.006Z |          |    8: instance_start.ensure_local_storage_2 | succeeded     | 
2026-04-03T18:10:12.009Z |          |    7: instance_start.ensure_local_storage_1 | started       | 
2026-04-03T18:10:14.693Z |          |    7: instance_start.ensure_local_storage_1 | succeeded     | 
2026-04-03T18:19:27.911Z |          |    6: instance_start.ensure_local_storage_0 | succeeded     | 
2026-04-03T18:19:27.919Z |          |               18: instance_start.dpd_ensure | started       | 
2026-04-03T18:19:27.936Z |          |               18: instance_start.dpd_ensure | failed        | "dpd_ensure" => {"ActionFailed":{"source_error":{"ObjectNotFound":{"lookup_type":{"ById":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa5
9"},"type_name":"instance"}}}}
2026-04-03T18:19:28.045Z |          |    8: instance_start.ensure_local_storage_2 | undo_started  | 
2026-04-03T18:19:28.048Z |          |    8: instance_start.ensure_local_storage_2 | undo_finished | 
2026-04-03T18:19:28.052Z |          |    7: instance_start.ensure_local_storage_1 | undo_started  | 
2026-04-03T18:19:28.056Z |          |    7: instance_start.ensure_local_storage_1 | undo_finished | 
2026-04-03T18:19:28.075Z |          |    6: instance_start.ensure_local_storage_0 | undo_started  | 
2026-04-03T18:19:28.079Z |          |    6: instance_start.ensure_local_storage_0 | undo_finished | 
2026-04-03T18:19:28.088Z |          |    9: instance_start.ensure_local_storage_3 | undo_started  | 
2026-04-03T18:19:28.091Z |          |    9: instance_start.ensure_local_storage_3 | undo_finished | 
2026-04-03T18:19:28.119Z |          |   10: instance_start.ensure_local_storage_4 | undo_started  | 
2026-04-03T18:19:28.122Z |          |   10: instance_start.ensure_local_storage_4 | undo_finished | 
2026-04-03T18:19:28.126Z |          |   11: instance_start.ensure_local_storage_5 | undo_started  | 
2026-04-03T18:19:28.129Z |          |   11: instance_start.ensure_local_storage_5 | undo_finished | 
2026-04-03T18:19:28.132Z |          |   12: instance_start.ensure_local_storage_6 | undo_started  | 
2026-04-03T18:19:28.135Z |          |   12: instance_start.ensure_local_storage_6 | undo_finished | 
2026-04-03T18:19:28.139Z |          |   13: instance_start.ensure_local_storage_7 | undo_started  | 
2026-04-03T18:19:28.142Z |          |   13: instance_start.ensure_local_storage_7 | undo_finished | 
2026-04-03T18:19:28.145Z |          |   14: instance_start.ensure_local_storage_8 | undo_started  | 
2026-04-03T18:19:28.148Z |          |   14: instance_start.ensure_local_storage_8 | undo_finished | 
2026-04-03T18:19:28.150Z |          |   15: instance_start.ensure_local_storage_9 | undo_started  | 
2026-04-03T18:19:28.153Z |          |   15: instance_start.ensure_local_storage_9 | undo_finished | 
2026-04-03T18:19:28.156Z |          |  16: instance_start.ensure_local_storage_10 | undo_started  | 
2026-04-03T18:19:28.186Z |          |  16: instance_start.ensure_local_storage_10 | undo_finished | 
2026-04-03T18:19:28.193Z |          |  17: instance_start.ensure_local_storage_11 | undo_started  | 
2026-04-03T18:19:28.197Z |          |  17: instance_start.ensure_local_storage_11 | undo_finished | 
2026-04-03T18:19:28.201Z |          |        5: instance_start.list_local_storage | undo_started  | 
2026-04-03T18:19:28.205Z |          |        5: instance_start.list_local_storage | undo_finished | 
2026-04-03T18:19:28.209Z |          |          4: instance_start.mark_as_starting | undo_started  | 
2026-04-03T18:19:28.270Z |          |          4: instance_start.mark_as_starting | undo_finished | 
2026-04-03T18:19:28.273Z |          |         3: instance_start.create_vmm_record | undo_started  | 
2026-04-03T18:19:28.281Z |          |         3: instance_start.create_vmm_record | undo_finished | 
2026-04-03T18:19:28.284Z |          |         2: instance_start.alloc_propolis_ip | undo_started  | 
2026-04-03T18:19:28.305Z |          |         2: instance_start.alloc_propolis_ip | undo_finished | 
2026-04-03T18:19:28.326Z |          |              1: instance_start.alloc_server | undo_started  | 
2026-04-03T18:19:28.335Z |          |              1: instance_start.alloc_server | undo_finished | 
2026-04-03T18:19:28.342Z |          |      0: instance_start.generate_propolis_id | undo_started  | 
2026-04-03T18:19:28.346Z |          |      0: instance_start.generate_propolis_id | undo_finished | 
2026-04-03T18:19:28.352Z |          |                                   24: start | undo_started  | 
2026-04-03T18:19:28.355Z |          |                                   24: start | undo_finished |

And the instance-delete saga:

 id                                   | current_sec                          | time_created             | name            | state 
--------------------------------------+--------------------------------------+--------------------------+-----------------+-------
 a253b289-4105-449e-bd99-ddaa5ec94ae5 | f3f20c9b-eb7b-4540-8eda-ac051b7f2c22 | 2026-04-03T18:10:39.141Z | instance-delete | Done  

DAG: {"end_node":7,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[6,0,null],[5,7,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"instance_delete.instance_delete_dendrite_config","label":"InstanceDeleteDendriteConfig","name":"no_result6"}},{"Action":{"action_name":"instance_delete.instance_delete_record","label":"InstanceDeleteRecord","name":"no_result1"}},{"Action":{"action_name":"instance_delete.delete_network_interfaces","label":"DeleteNetworkInterfaces","name":"no_result2"}},{"Action":{"action_name":"instance_delete.deallocate_external_ip","label":"DeallocateExternalIp","name":"no_result3"}},{"Action":{"action_name":"instance_delete.detach_external_subnets","label":"DetachExternalSubnets","name":"no_result4"}},{"Action":{"action_name":"instance_delete.leave_multicast_groups","label":"LeaveMulticastGroups","name":"no_result5"}},{"Start":{"params":{"authz_instance":{"key":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","lookup_type":{"ByName":"test-instance-for-disks"},"parent":{"key":"741c3913-77a7-46c7-8600-3cddc1737ccb","lookup_type":{"ByName":"abba"},"parent":{"key":"074fc33d-ef0b-46e9-ae15-d45b0fffba01","lookup_type":{"ById":"074fc33d-ef0b-46e9-ae15-d45b0fffba01"},"parent":null}}},"boundary_switches":["switch0","switch1"],"instance":{"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":"89d3e9f6-2029-4dde-8b9d-a6a94ecf4d90","cpu_platform":null,"hostname":"test-host","identity":{"description":"test-instance","id":"ac0ea3b7-4ef3-4cd1-9e59-8995347daa59","name":"test-instance-for-disks","time_created":"2026-04-03T18:06:36.579290Z","time_deleted":null,"time_modified":"2026-04-03T18:10:26.375778Z"},"intended_state":"Stopped","memory":17179869184,"ncpus":4,"project_id":"741c3913-77a7-46c7-8600-3cddc1737ccb","runtime_state":{"dst_propolis_id":null,"gen":7,"migration_id":null,"nexus_state":"NoVmm","propolis_id":null,"time_last_auto_restarted":null,"time_updated":"2026-04-03T18:10:28.548362Z"},"updater_gen":7,"updater_id":null,"user_data":[]},"serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"074fc33d-ef0b-46e9-ae15-d45b0fffba01","silo_user_id":"ebe3b1f0-634a-4ebb-82ef-991418f4e7da"}},"credential_id":"fa46b61f-27f5-46ce-8c88-76fdc178382e","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"End"]},"saga_name":"instance-delete","start_node":6}

              event time | sub saga | node id                                              | event type | data
------------------------ | -------- | ---------------------------------------------------- | ---------- | ---
2026-04-03T18:10:39.200Z |          |                                             6: start | started    | 
2026-04-03T18:10:39.212Z |          |                                             6: start | succeeded  | 
2026-04-03T18:10:39.217Z |          |   0: instance_delete.instance_delete_dendrite_config | started    | 
2026-04-03T18:10:39.452Z |          |   0: instance_delete.instance_delete_dendrite_config | succeeded  | 
2026-04-03T18:10:39.469Z |          |            1: instance_delete.instance_delete_record | started    | 
2026-04-03T18:10:39.594Z |          |            1: instance_delete.instance_delete_record | succeeded  | 
2026-04-03T18:10:39.604Z |          |         2: instance_delete.delete_network_interfaces | started    | 
2026-04-03T18:10:39.634Z |          |         2: instance_delete.delete_network_interfaces | succeeded  | 
2026-04-03T18:10:39.659Z |          |            3: instance_delete.deallocate_external_ip | started    | 
2026-04-03T18:10:39.888Z |          |            3: instance_delete.deallocate_external_ip | succeeded  | 
2026-04-03T18:10:39.896Z |          |           4: instance_delete.detach_external_subnets | started    | 
2026-04-03T18:10:39.906Z |          |           4: instance_delete.detach_external_subnets | succeeded  | 
2026-04-03T18:10:39.910Z |          |            5: instance_delete.leave_multicast_groups | started    | 
2026-04-03T18:10:39.914Z |          |            5: instance_delete.leave_multicast_groups | succeeded  | 
2026-04-03T18:10:39.917Z |          |                                               7: end | started    | 
2026-04-03T18:10:39.952Z |          |                                               7: end | succeeded  |

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/src/app/sagas/instance_start.rs around lines 700-702, where local-storage creation is polled with POST requests to sled-agent. Trace how the saga handles repeated 503 responses and deletion between polls; done means a deleted allocation is not recreated while the instance-start saga is still polling.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.