disk shakka-14e36227-0984-484e-8c94-baa-a88a42 on dogfood has problems

While debugging another issue, we found that the disk named
shakka-14e36227-0984-484e-8c94-baa-a88a42
with ID d2854af1-3397-4083-bd56-17b85bca6414 has a bad/corrupted volume.

From chat log with @jmpesp

root@[fd00:1122:3344:105::3]:32221/omicron> select region.* from region join crucible_dataset on crucible_dataset.id = region.dataset_id where crucible_dataset.ip = 'fd00:1122:3344:104::9' and region.port = 19015;
                   id                  |         time_created          |         time_modified         |              dataset_id              |              volume_id               | block_size | blocks_per_extent | extent_count | port  | read_only | deleting
---------------------------------------+-------------------------------+-------------------------------+--------------------------------------+--------------------------------------+------------+-------------------+--------------+-------+-----------+-----------
  131b0034-74f0-45de-b2e8-44c1a7c34a60 | 2024-02-15 05:05:56.737709+00 | 2024-02-15 05:05:56.737709+00 | c6fde82d-8dae-4ef0-b557-6c3d094d9454 | c2292d2e-fba5-4763-8ca0-9650a8e669a8 |        512 |            131072 |          160 | 19015 |   false   |  false
(1 row)


Time: 7ms total (execution 7ms / network 0ms)

then

root@[fd00:1122:3344:105::3]:32221/omicron> select * from region_snapshot where snapshot_addr = '[fd00:1122:3344:104::9]:19015';
               dataset_id              |              region_id               |             snapshot_id              |         snapshot_addr         | volume_references | deleting
---------------------------------------+--------------------------------------+--------------------------------------+-------------------------------+-------------------+-----------
  c6fde82d-8dae-4ef0-b557-6c3d094d9454 | ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171 | e35f28da-05fb-4f36-a8fe-514eb9708867 | [fd00:1122:3344:104::9]:19015 |                13 |  false
(1 row)

the fact that it has a non-zero volume references count means it's old I think

root@[fd00:1122:3344:105::3]:32221/omicron> select * from region_snapshot where snapshot_addr = '[fd00:1122:3344:104::9]:19015';
               dataset_id              |              region_id               |             snapshot_id              |         snapshot_addr         | volume_references | deleting
---------------------------------------+--------------------------------------+--------------------------------------+-------------------------------+-------------------+-----------
  c6fde82d-8dae-4ef0-b557-6c3d094d9454 | ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171 | e35f28da-05fb-4f36-a8fe-514eb9708867 | [fd00:1122:3344:104::9]:19015 |                13 |  false
(1 row)


Time: 3ms total (execution 3ms / network 0ms)

root@[fd00:1122:3344:105::3]:32221/omicron> select * from volume_resource_usage where region_snapshot_dataset_id = 'c6fde82d-8dae-4ef0-b557-6c3d094d9454' and region_snapshot_region_id = 'ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171' and region_snapshot_snapshot_id = 'e35f28da-05fb-4f36-a8fe-514eb9708867';
                usage_id               |              volume_id               |   usage_type    | region_id |      region_snapshot_dataset_id      |      region_snapshot_region_id       |     region_snapshot_snapshot_id
---------------------------------------+--------------------------------------+-----------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------
  8527e916-e013-46a4-ba1d-93666694acb1 | 40c187c6-ba33-43f0-a2e9-493b315f8729 | region_snapshot | NULL      | c6fde82d-8dae-4ef0-b557-6c3d094d9454 | ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171 | e35f28da-05fb-4f36-a8fe-514eb9708867
  dfd5faae-3a73-4cc1-a9bf-e8a082703750 | b93a0e45-3e55-4a0f-8923-7d84b8552be7 | region_snapshot | NULL      | c6fde82d-8dae-4ef0-b557-6c3d094d9454 | ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171 | e35f28da-05fb-4f36-a8fe-514eb9708867
  f372f447-9a52-4a84-b6ff-62ed92c86338 | ad2b7942-2365-40d8-b193-a796dc4e9feb | region_snapshot | NULL      | c6fde82d-8dae-4ef0-b557-6c3d094d9454 | ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171 | e35f28da-05fb-4f36-a8fe-514eb9708867
(3 rows)

or didn't get cleaned up properly?

BRM44220005 # curl -sqSL 'http://[fd00:1122:3344:104::9]:32345/
crucible
/0/regions/131b0034-74f0-45de-b2e8-44c1a7c34a60' | jq .
{
  "id": "131b0034-74f0-45de-b2e8-44c1a7c34a60",
  "state": "destroyed",
  "block_size": 512,
  "extent_size": 131072,
  "extent_count": 160,
  "encrypted": true,
  "port_number": 19015,
  "cert_pem": null,
  "key_pem": null,
  "root_pem": null,
  "source": null,
  "read_only": false
}

destroyed - this is fishy

BRM44220005 # curl -sqSL 'http://[fd00:1122:3344:104::9]:32345/
crucible
/0/regions/ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171/snapshots' | jq .
{
  "snapshots": [
    {
      "name": "e35f28da-05fb-4f36-a8fe-514eb9708867",
      "created": "2024-11-06T01:48:34Z"
    }
  ],
  "running_snapshots": {
    "e35f28da-05fb-4f36-a8fe-514eb9708867": {
      "id": "ef7e5cdb-5f33-4c9b-b9dd-6ec96424f171",
      "name": "e35f28da-05fb-4f36-a8fe-514eb9708867",
      "port_number": 19015,
      "state": "created"
    }
  }
}

the snapshot's the real one here

 dataset_id                            region_id                             dataset_addr                   error                                       
 f9c1deca-1898-429e-8c93-254c7aa7bae6  038ed251-9bbf-47c1-8eae-409de7957709  [fd00:1122:3344:105::8]:32345  Agent does not know about this region!      
 f9c1deca-1898-429e-8c93-254c7aa7bae6  0ef5add9-408f-498c-b3f4-e8d625b40c52  [fd00:1122:3344:105::8]:32345  Agent does not know about this region!      
 c6fde82d-8dae-4ef0-b557-6c3d094d9454  131b0034-74f0-45de-b2e8-44c1a7c34a60  [fd00:1122:3344:104::9]:32345  region may need to be manually hard-deleted <-- our problematic one from `cooked`
 a6ba8273-0320-4dab-b801-281f041b0c50  3e275398-08ac-4255-9612-f3ed3a8e8358  [fd00:1122:3344:103::4]:32345  region may need to be manually hard-deleted 
 0022703b-dcfc-44d4-897a-b42f6f53b433  45751d83-86d1-476f-866e-79d3a72c32f1  [fd00:1122:3344:106::c]:32345  region may need to be manually hard-deleted 
 23dca27d-c79b-4930-a817-392e8aeaa4c1  6b07a11b-60cf-43ac-8827-909f00099675  [fd00:1122:3344:105::e]:32345  Agent does not know about this region!      
 f65a6668-1aea-4deb-81ed-191fbe469328  5019a6a4-ea05-445c-a526-7ff1e6fd7ec0  [fd00:1122:3344:107::9]:32345  region may need to be manually hard-deleted 
 56d4dbcc-3b4a-4ed0-8795-7734aadcc4c0  5448ae38-9263-459e-9459-aef3efa925bd  [fd00:1122:3344:101::9]:32345  region may need to be manually hard-deleted 
 23dca27d-c79b-4930-a817-392e8aeaa4c1  d9968d58-5e4f-4349-aaf9-f88024ebf8b4  [fd00:1122:3344:105::e]:32345  Agent does not know about this region!      
 0022703b-dcfc-44d4-897a-b42f6f53b433  df0da3c6-8629-4aea-b677-6414b6b1177c  [fd00:1122:3344:106::c]:32345  region may need to be manually hard-deleted 
 d4d9acc8-3e0b-4fab-a0a2-d21920fabd7e  b4390003-af2d-462c-8025-15c38588cd97  [fd00:1122:3344:10a::6]:32345  region may need to be manually hard-deleted 
 90b53c3d-42fa-4ca9-bbfc-96fff245b508  d270fae8-2cba-4a52-b07c-8b222e3f8edb  [fd00:1122:3344:109::4]:32345  region may need to be manually hard-deleted 
found 0 orphaned bytes

omdb db validate validate-regions catches this
I guess we need a reconciler?
there's one volume that may be using it, as it's a read-write region set:

{
  "type": "volume",
  "block_size": 512,
  "id": "d2854af1-3397-4083-bd56-17b85bca6414",
  "sub_volumes": [
    {
      "type": "region",
      "block_size": 512,
      "blocks_per_extent": 131072,
      "extent_count": 1600,
      "gen": 2,
      "opts": {
        "id": "d2854af1-3397-4083-bd56-17b85bca6414",
        "key": "3xXi8wqw6U6aMh2PUPePnGLvdfXHDxG2X0Jkov2FepU=",
        "lossy": false,
        "read_only": false,
        "target": [
          "[fd00:1122:3344:106::4]:19003",
          "[fd00:1122:3344:108::e]:19026",
          "[fd00:1122:3344:102::7]:19012"
        ]
      }
    }
  ]
}
0000b90f-6548-42ef-bcac-4733c6740180

root@[fd00:1122:3344:105::3]:32221/omicron> select * from disk where volume_id = '0000b90f-6548-42ef-bcac-4733c6740180';
id | name | description | time_created | time_modified | time_deleted | rcgen | project_id | volume_id | disk_state | attach_instance_id | state_generation | slot | time_state_updated | size_bytes | block_size | origin_snapshot | origin_image | pantry_address
---------------------------------------+-------------------------------------------+-----------------------------------+-------------------------------+-------------------------------+--------------+-------+--------------------------------------+--------------------------------------+------------+--------------------------------------+------------------+------+-------------------------------+--------------+------------+-----------------+--------------------------------------+-----------------
d2854af1-3397-4083-bd56-17b85bca6414 | shakka-14e36227-0984-484e-8c94-baa-a88a42 | Created as a boot disk for shakka | 2024-08-15 23:39:33.243956+00 | 2024-08-15 23:39:33.243956+00 | NULL | 1 | 57c1a4d2-920f-4577-b6f5-6542d07dfb1b | 0000b90f-6548-42ef-bcac-4733c6740180 | attached | c572c155-b109-439a-ab41-95fe6711c6f4 | 2 | 0 | 2024-08-15 23:39:36.074315+00 | 107374182400 | 512 | NULL | 14e36227-0984-484e-8c94-baab1a6be648 | NULL
(1 row)

there's no way this disk works

root@[fd00:1122:3344:105::3]:32221/omicron> select * from instance where id = 'c572c155-b109-439a-ab41-95fe6711c6f4';
id | name | description | time_created | time_modified | time_deleted | project_id | user_data | time_state_updated | state_generation | active_propolis_id | target_propolis_id | migration_id | ncpus | memory | hostname | updater_id | updater_gen | state | time_last_auto_restarted | auto_restart_policy | auto_restart_cooldown | boot_disk_id
---------------------------------------+--------+-------------+-------------------------------+-------------------------------+--------------+--------------------------------------+-----------+------------------------------+------------------+--------------------+--------------------+--------------+-------+-------------+----------+------------+-------------+--------+--------------------------+---------------------+-----------------------+---------------------------------------
c572c155-b109-439a-ab41-95fe6711c6f4 | shakka | walls fell | 2024-08-15 23:39:31.305964+00 | 2024-08-15 23:39:31.305964+00 | NULL | 57c1a4d2-920f-4577-b6f5-6542d07dfb1b | | 2024-08-21 19:10:17.47581+00 | 4 | NULL | NULL | NULL | 4 | 17179869184 | shakka | NULL | 4 | no_vmm | NULL | NULL | NULL | d2854af1-3397-4083-bd56-17b85bca6414
(1 row)

Start with the omdb db validate validate-regions entry point and the region, region_snapshot, and volume_resource_usage records shown in the report. Trace the relevant reconciliation or cleanup path, then define done as the corrupted region and its stale references being handled consistently without breaking the disk's volume relationship.