sled-agent failure to create zpools can wedge RSS

Today on madrid, RSS running on sled 16 got stuck with this error repeating trying to set up internal DNS on sled 17:

01:02:27.442Z WARN SledAgent (RSS): failed to initialize Omicron zones
    error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2fd3894f-959c-4517-b352-4ca4752a793a", "content-length": "124", "date": "Sun, 28 Dec 1986 00:34:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2fd3894f-959c-4517-b352-4ca4752a793a" }
    file = sled-agent/src/rack_setup/service.rs:467
    retry_after = 122.524448892s
    sled_address = [fd00:1122:3344:101::1]:12345
01:04:29.968Z INFO SledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 53fc85a2-46e2-463d-a1ca-816b88b865de, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89a8ee2c-6299-44bd-a614-cd0bc453d126 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }] }
    file = sled-agent/src/rack_setup/service.rs:429
    sled_address = [fd00:1122:3344:101::1]:12345
01:04:29.968Z WARN SledAgent (RSS): failed to initialize Omicron zones
    error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e5f0426e-2dc4-4d89-ae2b-4874a35747c6", "content-length": "124", "date": "Sun, 28 Dec 1986 00:36:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e5f0426e-2dc4-4d89-ae2b-4874a35747c6" }
    file = sled-agent/src/rack_setup/service.rs:467
    retry_after = 250.175406396s
    sled_address = [fd00:1122:3344:101::1]:12345

Looking for this error on sled 17, we see that the zpool RSS wants to use for internal DNS doesn't exist:

00:34:33.560Z INFO SledAgent (dropshot (SledAgent)): request completed
    error_message_external = Internal Server Error
    error_message_internal = Error managing storage: Zpool Not Found: oxp_89a8ee2c-6299-44bd-a614-cd0bc453d126
    file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837
    latency_us = 137
    local_addr = [fd00:1122:3344:101::1]:12345
    method = PUT
    remote_addr = [fd00:1122:3344:102::1]:63777
    req_id = 2fd3894f-959c-4517-b352-4ca4752a793a
    response_code = 500
    uri = /omicron-zones

Earlier in the RSS process, we told this sled to use 10 disks, including the above zpool:

00:39:15.082Z INFO SledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: e62413b2-eba6-4fd2-9a37-459dd8c4c6ec, identity: DiskIdentity { vendor: "1b96", serial: "A079DE8D", model: "WUS4C6432DSP3X3" }, pool_id: 89a8ee2c-6299-44bd-a614-cd0bc453d126 (zpool) }, OmicronPhysicalDiskConfig { id: 2f9d6da8-17f6-45ba-b4d5-d3675466c053, identity: DiskIdentity { vendor: "1b96", serial: "A079DEA6", model: "WUS4C6432DSP3X3" }, pool_id: 2fefc812-cf06-4758-bca5-2da3cea3020e (zpool) }, OmicronPhysicalDiskConfig { id: 6d472bec-c965-49ed-ac8c-1ae4c9ab3428, identity: DiskIdentity { vendor: "1b96", serial: "A079E1B3", model: "WUS4C6432DSP3X3" }, pool_id: 67250dfa-4bdb-4293-8424-7688a73ed01b (zpool) }, OmicronPhysicalDiskConfig { id: 8a7679f0-cbf8-49da-8c68-90d4461226fa, identity: DiskIdentity { vendor: "1b96", serial: "A079E3F3", model: "WUS4C6432DSP3X3" }, pool_id: f16c8ae7-1531-4b5d-9a8f-5f88010cda79 (zpool) }, OmicronPhysicalDiskConfig { id: baeb5221-fb46-4926-83ef-27e256bd9f97, identity: DiskIdentity { vendor: "1b96", serial: "A079E3F8", model: "WUS4C6432DSP3X3" }, pool_id: dcc22060-e50e-4e0f-8384-f1c72cd0e59b (zpool) }, OmicronPhysicalDiskConfig { id: abf7324b-b8d8-42f7-b008-3e6ca1f57a5a, identity: DiskIdentity { vendor: "1b96", serial: "A079E4AA", model: "WUS4C6432DSP3X3" }, pool_id: 4246dabe-2b8b-4db7-8606-738ff4292f24 (zpool) }, OmicronPhysicalDiskConfig { id: 87f0951c-4941-4049-bd6c-1c8f10078039, identity: DiskIdentity { vendor: "1b96", serial: "A079E4C6", model: "WUS4C6432DSP3X3" }, pool_id: 47d00217-ccb0-4655-966c-1369478796a9 (zpool) }, OmicronPhysicalDiskConfig { id: 83569af4-d7f7-497d-8c8b-1a234fa06854, identity: DiskIdentity { vendor: "1b96", serial: "A079E53D", model: "WUS4C6432DSP3X3" }, pool_id: 436ffe85-94e9-4399-9926-0eba22772393 (zpool) }, OmicronPhysicalDiskConfig { id: 7a5ac4e4-4078-4bce-a653-95e790b43ce1, identity: DiskIdentity { vendor: "1b96", serial: "A079E576", model: "WUS4C6432DSP3X3" }, pool_id: efddfa96-23b5-4f47-8137-d706338329e4 (zpool) }, OmicronPhysicalDiskConfig { id: 110c139d-404b-474f-a1e1-ad5b581ef787, identity: DiskIdentity { vendor: "1b96", serial: "A084A733", model: "WUS4C6432DSP3X3" }, pool_id: 7840d699-6cc5-4d59-8a9e-247e1253e30c (zpool) }], generation: Generation(1) }
    file = sled-agent/src/rack_setup/service.rs:346
    sled_address = [fd00:1122:3344:101::1]:12345

Finding this request on sled 17, we see it succeeded:

00:11:21.306Z INFO SledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079DE8D", model: "WUS4C6432DSP3X3" }, id: e62413b2-eba6-4fd2-9a37-459dd8c4c6ec, pool_id: 89a8ee2c-6299-44bd-a614-cd0bc453d126 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079DEA6", model: "WUS4C6432DSP3X3" }, id: 2f9d6da8-17f6-45ba-b4d5-d3675466c053, pool_id: 2fefc812-cf06-4758-bca5-2da3cea3020e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E1B3", model: "WUS4C6432DSP3X3" }, id: 6d472bec-c965-49ed-ac8c-1ae4c9ab3428, pool_id: 67250dfa-4bdb-4293-8424-7688a73ed01b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E3F3", model: "WUS4C6432DSP3X3" }, id: 8a7679f0-cbf8-49da-8c68-90d4461226fa, pool_id: f16c8ae7-1531-4b5d-9a8f-5f88010cda79 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E3F8", model: "WUS4C6432DSP3X3" }, id: baeb5221-fb46-4926-83ef-27e256bd9f97, pool_id: dcc22060-e50e-4e0f-8384-f1c72cd0e59b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E4AA", model: "WUS4C6432DSP3X3" }, id: abf7324b-b8d8-42f7-b008-3e6ca1f57a5a, pool_id: 4246dabe-2b8b-4db7-8606-738ff4292f24 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E4C6", model: "WUS4C6432DSP3X3" }, id: 87f0951c-4941-4049-bd6c-1c8f10078039, pool_id: 47d00217-ccb0-4655-966c-1369478796a9 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E53D", model: "WUS4C6432DSP3X3" }, id: 83569af4-d7f7-497d-8c8b-1a234fa06854, pool_id: 436ffe85-94e9-4399-9926-0eba22772393 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A079E576", model: "WUS4C6432DSP3X3" }, id: 7a5ac4e4-4078-4bce-a653-95e790b43ce1, pool_id: efddfa96-23b5-4f47-8137-d706338329e4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "1b96", serial: "A084A733", model: "WUS4C6432DSP3X3" }, id: 110c139d-404b-474f-a1e1-ad5b581ef787, pool_id: 7840d699-6cc5-4d59-8a9e-247e1253e30c (zpool) }] }, tx: ... }
    file = sled-storage/src/manager.rs:381
00:11:21.306Z INFO SledAgent (StorageManager): No ledger in /pool/int/ad41e0a6-120f-4475-ac8b-9b11b20b4a10/config/omicron-physical-disks.json
    file = common/src/ledger.rs:188
    request = omicron_physical_disks_ensure
00:11:21.306Z INFO SledAgent (StorageManager): No ledger in /pool/int/723cf5be-f05d-4894-a5bf-66451062ab03/config/omicron-physical-disks.json
    file = common/src/ledger.rs:188
    request = omicron_physical_disks_ensure
00:11:21.306Z INFO SledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger
    file = sled-storage/src/manager.rs:681
    request = omicron_physical_disks_ensure
00:11:51.239Z INFO SledAgent (StorageManager): Writing ledger to /pool/int/ad41e0a6-120f-4475-ac8b-9b11b20b4a10/config/.omicron-physical-disks.json.tmp
    file = common/src/ledger.rs:199
    request = omicron_physical_disks_ensure
00:11:51.395Z INFO SledAgent (StorageManager): Writing ledger to /pool/int/723cf5be-f05d-4894-a5bf-66451062ab03/config/.omicron-physical-disks.json.tmp
    file = common/src/ledger.rs:199
    request = omicron_physical_disks_ensure
00:11:51.396Z INFO SledAgent (dropshot (SledAgent)): request completed
    file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849
    latency_us = 30091613
    local_addr = [fd00:1122:3344:101::1]:12345
    method = PUT
    remote_addr = [fd00:1122:3344:102::1]:63122
    req_id = 88c7e615-0232-485f-ae40-520742b4aa23
    response_code = 200
    uri = /omicron-physical-disks

However, the sled-agent on sled17 was NOT able to create the zpools on 2 of these 10 disks:

00:11:21.553Z WARN SledAgent (StorageResources): Disk::new failed
    err = PooledDisk(ZpoolCreate(CreateError { err: Execution(CommandFailure(CommandFailureInfo { command: "/usr/sbin/zpool create -o ashift=12 oxp_89a8ee2c-6299-44bd-a614-cd0bc453d126 /devices/pci@38,0/pci1de,fff9@1,2/pci1b96,0@0/blkdev@w0014EE81000BC129,0:a", status: ExitStatus(unix_wait_status(256)), stdout: "", stderr: "cannot open '/devices/pci@38,0/pci1de,fff9@1,2/pci1b96,0@0/blkdev@w0014EE81000BC129,0:a': No such device or address\\n" })) }))
    file = sled-storage/src/resources.rs:466
00:11:24.832Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079DEA6", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:28.025Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E1B3", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:31.432Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E3F3", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:31.433Z WARN SledAgent (StorageResources): Disk::new failed
    err = PooledDisk(BadPartitionLayout { path: "/devices/pci@ab,0/pci1de,fff9@1,2/pci1b96,0@0/blkdev@w0014EE81000BC440,0:wd,raw", why: "Expected 1 partitions, only saw 0" })
    file = sled-storage/src/resources.rs:466
00:11:35.111Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E4AA", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:38.924Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E4C6", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:43.388Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E53D", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:47.432Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A079E576", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477
00:11:51.239Z INFO SledAgent (StorageResources): Disk::new completed successfully
    disk_identity = DiskIdentity { vendor: "1b96", serial: "A084A733", model: "WUS4C6432DSP3X3" }
    file = sled-storage/src/resources.rs:477

This is a known problem with these two disks on this sled (#5128); however, new behavior causes this to block RSS from completing. IIUC, before #5172, RSS would ask the sled-agents for their zpools, and then select from them when assigning zpools to services. This meant that failures to create zpools (such as the above) meant those disks would never be chosen for services. However, after #5172, RSS asks the sled-agents for inventory; sled-agent doesn't actually try to create zpools until it's instructed to (by RSS or Nexus later). The inventory returns all 10 physical disks, so RSS assumes it is free to select any of them for control plane services, leading to the wedge above where it chooses a disk where zpool creation fails, meaning RSS can't make forward progress.

There are (at least) two issues here; one is #5128: we don't believe there's anything wrong with these two disks at the hardware level, so sled-agent ought to be able to reformat them (probably? some questions here about policy and operator intervention, etc.). The other is that there's no way for RSS to recover if sled-agent fails to create a zpool for any of the disks fails for any unrecoverable reason.

Start with sled-storage/src/resources.rs and sled-agent/src/rack_setup/service.rs, then compare the failed disk and zpool handling with issue #5128. Trace how partial zpool creation is reported through the storage manager and rack setup path; done means a failed disk setup does not leave RSS repeatedly unable to initialize its zones.