`omdb disks info` is misreporting dataset IDs instead of zone IDs for regions.

Here is some omdb output. I will explain it after.

root@oxz_switch1:~# omdb db disks info 341d9b3e-bf37-40f9-9dfc-8304abd241f8
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:103::3]:32221,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:104::3]:32221,[fd00:1122:3344:104::4]:32221,[fd00:1122:3344:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (114.0.0)
HOST_SERIAL DISK_NAME          INSTANCE_NAME           PROPOLIS_ZONE                                            VOLUME_ID                            DISK_STATE 
BRM42220036 iodrvtgt-x-15810-1 iodriver-worker-15810-1 oxz_propolis-server_84b10e54-2e1c-4839-b6bc-699139de793b a21e3724-9721-40e9-bd90-dfc6169487e1 attached   
HOST_SERIAL REGION                               ZONE                                              PHYSICAL_DISK                        
BRM44220007 eab55cb7-b146-426c-a16a-67c32e455832 oxz_crucible_08c87647-fc7a-427b-a831-6feac2cae153 52f73ef4-f6bf-463b-91c3-fa28b4cc0f8d 
BRM42220036 f6740bf9-2404-49dc-b1a0-192a14357ef0 oxz_crucible_240477f5-7c26-4eb7-8377-407f5325d19d ef67ab58-b6b6-4828-b648-07b306e3e473 
BRM42220030 a1d7cbe7-528c-4c42-925e-adf64c0d8efb oxz_crucible_92b86613-cfde-4350-af57-b72e96faea4d d9ca468b-b9f6-4610-88cf-832b07ea763a 

VCR from volume ID a21e3724-9721-40e9-bd90-dfc6169487e1
ID                                   BS   SUB_VOLUMES READ_ONLY_PARENT 
341d9b3e-bf37-40f9-9dfc-8304abd241f8 4096 1           false            

SUB VOLUME 0
    ID                                   BS   BPE   EC   GEN READ_ONLY 
    341d9b3e-bf37-40f9-9dfc-8304abd241f8 4096 16384 1024 2   false     
    [fd00:1122:3344:101::8]:19007
    [fd00:1122:3344:102::8]:19004
    [fd00:1122:3344:104::e]:19003

Here's the part we care about:

HOST_SERIAL REGION                               ZONE                                              PHYSICAL_DISK                        
BRM44220007 eab55cb7-b146-426c-a16a-67c32e455832 oxz_crucible_08c87647-fc7a-427b-a831-6feac2cae153 52f73ef4-f6bf-463b-91c3-fa28b4cc0f8d 
BRM42220036 f6740bf9-2404-49dc-b1a0-192a14357ef0 oxz_crucible_240477f5-7c26-4eb7-8377-407f5325d19d ef67ab58-b6b6-4828-b648-07b306e3e473 
BRM42220030 a1d7cbe7-528c-4c42-925e-adf64c0d8efb oxz_crucible_92b86613-cfde-4350-af57-b72e96faea4d d9ca468b-b9f6-4610-88cf-832b07ea763a 

• HOST_SERIAL tells us the sled a disk is on.
• REGION is the crucible region ID.
• ZONE should be (and historically, has been) the zone on the sled which has the region in it.
  • Note that one crucible zone has many regions in it. We have one crucible zone per physical u.2 device.
• PHYSICAL_DISK is the UUID of the u.2

The problem is the ZONEs reported here do not exist.

root@oxz_switch1:~# pilot host exec -c 'zoneadm list | grep oxz_crucible_240477f5-7c26-4eb7-8377-407f5325d19d' 1-31
14  BRM42220036        failure: exit code 1
15  BRM42220062        failure: exit code 1
16  BRM42220030        failure: exit code 1
17  BRM44220007        failure: exit code 1
Error: some operations failed

However the HOST_SERIAL and REGION are correct. So we can find the zones that these things are in:

root@oxz_switch1:~# omdb db disks info 341d9b3e-bf37-40f9-9dfc-8304abd241f8 2>/dev/null | perl -nE '
    # this script takes the sled & region out of the regions table, and searches
    # the sled for any files with that region's UUID in the filename
    if (/HOST_SERIAL REGION/../^$/ and /^BRM/) {
        my @cols = split;
        my $sled = $cols[0];
        my $region = $cols[1];
        system("pilot host exec -c \"find /pool/ext 2>/dev/null | grep ${region} | head -n1\" ${sled}");
    }
'
17  BRM44220007        ok: /pool/ext/7e51af96-029a-472d-b99a-8bc605dcdd04/crypt/zone/oxz_crucible_d9390738-774e-4941-9e00-11ee6f3de686/root/data/regions/eab55cb7-b146-426c-a16a-67c32e455832
14  BRM42220036        ok: /pool/ext/0482a241-c479-42bb-8354-b8e14559b894/crypt/zone/oxz_crucible_e5c005a8-1c0e-4048-b964-493c3f4a84f2/root/data/regions/f6740bf9-2404-49dc-b1a0-192a14357ef0
16  BRM42220030        ok: /pool/ext/872af973-4096-4843-b18f-4d947e85902c/crypt/zone/oxz_crucible_94a2fc45-8ee7-48b5-a399-9337e88e109c/root/data/regions/a1d7cbe7-528c-4c42-925e-adf64c0d8efb

Incorrect, reported zones, which do not seem to actually exist as far as I can find:

• oxz_crucible_08c87647-fc7a-427b-a831-6feac2cae153
• oxz_crucible_240477f5-7c26-4eb7-8377-407f5325d19d
• oxz_crucible_92b86613-cfde-4350-af57-b72e96faea4d

Correct zones, which do exist, and have real logs in them:

• oxz_crucible_d9390738-774e-4941-9e00-11ee6f3de686
• oxz_crucible_e5c005a8-1c0e-4048-b964-493c3f4a84f2
• oxz_crucible_94a2fc45-8ee7-48b5-a399-9337e88e109c

So... what is it then if not a zone? We'll check all the logs

omdb db disks info 341d9b3e-bf37-40f9-9dfc-8304abd241f8 2>/dev/null | perl -nE '
    if (/HOST_SERIAL REGION/../^$/ and /^BRM/) {
        my @cols = split;
        my $sled = $cols[0];
        my $zone = $cols[2];
        $zone =~ s/oxz_crucible_//;
        my $cmd = "/opt/oxide/oxlog/oxlog zones | xargs -n1 /opt/oxide/oxlog/oxlog logs --current --archived | xargs cat | grep ${zone} | head -n1 | looker";
        system("pilot host exec -c \"${cmd}\" ${sled}");
    }
'

This prints some big rust debug blobs I'm not going to drop in verbatim, but here's one of our mis-reported UUIDs, 08c87647-fc7a-427b-a831-6feac2cae153:

08c87647-fc7a-427b-a831-6feac2cae153 (dataset): DatasetConfig {
    id: 08c87647-fc7a-427b-a831-6feac2cae153 (dataset),
    name: DatasetName {
        pool_name: ZpoolName {
            id: 7e51af96-029a-472d-b99a-8bc605dcdd04 (zpool), kind: External
        }, kind: Crucible
    }
    inner: ...
}

I think this means they are dataset IDs.

Start by reproducing the issue with the omdb db disks info command and trace how its disk information output obtains the REGION and ZONE columns. Compare the reported values with the sled and region evidence in the issue. Done means the command reports actual Crucible zone IDs rather than dataset IDs while preserving the correct host and region values.