Sagas with extended retry loops may be undrainable during upgrade windows

(This is a derived from one of the issues described in https://github.com/oxidecomputer/colo/issues/102.)

During the R13 colo update window, we expunged a disk that had been physically removed from the rack many months earlier. This (indirectly) caused one of the Nexus instances to panic inside steno:

thread 'tokio-runtime-worker{"msg":"recovered saga","v":0,"name":"nexus","level":30,"time":"2025-02-24T22:08:05.934257132Z","hostname":"oxz_nexus_1cfdb5b6-e568-436a-a85f-7fecf1b8eef2","pid":27476,"saga_id":"e8752892-cdb5-49b1-81ba-e6b3e45ab75e","saga_name":"image-delete","component":"nexus","component":"ServerContext","name":"1cfdb5b6-e568-436a-a85f-7fecf1b8eef2","file":"nexus/saga-recovery/src/recovery.rs:474"}
' panicked at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/steno-0.4.1/src/saga_exec.rs:2146:32:
no ancestor called "freed_crucible_regions"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

A panic like this indicates that Nexus was trying to execute a saga that was created in an older software release, which steno does not support. We have some related issues:

• #2318 notes that we should drain sagas as a part of upgrade
• #6804 goes into more detail into what that might look like while we're still in a manual upgrade world

However, this case on colo seems more complicated. The saga in question was created several months ago and was still running:

SQLSTATE: 42703
root@[fd00:1122:3344:116::3]:32221/omicron> select id, time_created, name, saga_state from saga where saga_state = 'running';
                   id                  |         time_created          |     name     | saga_state
---------------------------------------+-------------------------------+--------------+-------------
  3c974c5c-1782-4c8c-8eec-fba087fa40ab | 2024-09-27 17:41:17.32426+00  | image-delete | running
  e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.430556+00 | image-delete | running
(2 rows)

so it's hard to imagine any kind of "quiesce and drain" procedure would have been sufficient. Both of these sagas were stuck in the volume_delete.delete_crucible_regions step before resuming and panicking in a later step:

                             saga id | event time                     | node id                                              | event type | data
------------------------------------ | ------------------------------ | ---------------------------------------------------- | ---------- | ---
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.437136 UTC |  11: start                                           | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.442525 UTC |  11: start                                           | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.445196 UTC |   0: image_delete.delete_image_record                | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.467926 UTC |   0: image_delete.delete_image_record                | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.470255 UTC |   1: delete_volume_params                            | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.472496 UTC |   1: delete_volume_params                            | succeeded  | {"serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"074fc33d-ef0b-46e9-ae15-d45b0fffba01","silo_user_id":"0449f6e1-d85c-4202-aabd-969c827ee457"}}},{"mapped_fleet_roles":{"admin":["admin"]}}]}},"volume_id":"b474dc3c-9987-4065-b077-7c650bde1165"}
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.474990 UTC |   2: subsaga start volume-delete                     | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.477164 UTC |   2: subsaga start volume-delete                     | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.479159 UTC |   3: volume_delete.decrease_crucible_resource_count  | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.501902 UTC |   3: volume_delete.decrease_crucible_resource_count  | succeeded  | {"V3":{"region_snapshots":[{"dataset":"41f07d39-fcc0-4796-8b7c-7cfcd9135f78","region":"1c133ade-86bc-4559-93fb-a2783784dc1b","snapshot":"e1ed98a4-39e1-4b20-8849-15bdd5ad3b2f"},{"dataset":"a109a902-6a27-41b6-a881-c353e28e5389","region":"904b64c4-7cf9-4b15-8ba4-8927bc0f84e6","snapshot":"e1ed98a4-39e1-4b20-8849-15bdd5ad3b2f"},{"dataset":"d00e1d0b-e12f-420a-a4df-21e4cac176f6","region":"97c58dab-6291-4b48-a689-a0e1265105c4","snapshot":"e1ed98a4-39e1-4b20-8849-15bdd5ad3b2f"}],"regions":[]}}
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.504284 UTC |   4: volume_delete.delete_crucible_regions           | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.506558 UTC |   5: volume_delete.delete_crucible_running_snapshots | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2024-09-27 17:36:43.509158 UTC |   4: volume_delete.delete_crucible_regions           | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.653744 UTC |   5: volume_delete.delete_crucible_running_snapshots | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.662157 UTC |   6: volume_delete.delete_crucible_snapshots         | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.820283 UTC |   6: volume_delete.delete_crucible_snapshots         | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.825366 UTC |   7: volume_delete.delete_crucible_snapshot_records  | started    | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.874770 UTC |   7: volume_delete.delete_crucible_snapshot_records  | succeeded  | 
e8752892-cdb5-49b1-81ba-e6b3e45ab75e | 2025-02-24 22:07:54.879216 UTC |   8: volume_delete.delete_freed_crucible_regions     | started    | 

I strongly suspect they were stuck in this ProgenitorOperationRetry, which is ultimately called by the volume_delete.delete_crucible_regions step: https://github.com/oxidecomputer/omicron/blob/8222b27e850d2c3adc496222a317fe5ff96d1289/nexus/src/app/crucible.rs#L460-L467

This operation retries until either:

• The Crucible agent in question responds to the region_get request
• crucible_agent_gone_check returns true, which internally checks "is the dataset for this Crucible zone still on an in-service disk"

If we were trying to check on a region for a dataset on the disk that had been removed many months ago, its Crucible agent would never respond, but because we hadn't gone through the expungement process, the dataset for it was still considered to be on an in-service disk. This would explain both why we were still running this saga after months (because we were waiting for one of the two above conditions, neither of which had happened yet) and why we suddenly made progress (because we expunged the disk, which caused crucible_agent_gone_check to return true) and then subsequently panicked (because now we tried to continue executing a saga from an old Nexus version, which we know isn't supported).

I'm not sure what to propose here. If we had instructions for operators to ensure sagas are drained prior to starting an upgrade, what should they do if a saga has been running for months?

Start with nexus/src/app/crucible.rs#L460-L467 and the linked saga-draining issues #2318 and #6804. Trace how ProgenitorOperationRetry can leave a saga running across an upgrade, then determine how long-running sagas should be handled. Done requires an agreed operator procedure or implementation that prevents the legacy-saga panic during upgrade windows.