sled-agent `physical_disk` inventory should include topo's `label`, rather than `pcieb`'s `physical-slot#`

Open
#11,258 5 comments 0 reactions 1 assignee View on GitHub

@smklein is already working on this.

Since Sep 9, 2026.

Assessment

This issue has not been assessed yet.

Description

fault-management rack hardware Sled Agent

See https://github.com/oxidecomputer/omicron/pull/11221#issuecomment-5574308943, https://github.com/oxidecomputer/omicron/pull/11221#issuecomment-5574551780, and https://github.com/oxidecomputer/omicron/pull/11221#pullrequestreview-5134670273.

As described in https://github.com/oxidecomputer/omicron/issues/11088, we would like to be able to include a slot number in the external API's physical_disk list and views. The value we return here needs to be the slot number marked on the chassis, since the intent behind such an API would be to allow an operator to identify where a disk is located within a sled for purposes such as disk replacement. Naively, I had written in #11088 that we could do this by using the inventory to look up a disk's sled by ID and then look up the slot in the inventory entry for that disk.

On Gimlets, the values for the slot field in the inventory's physical disk records are numbered 0-9 for U.2s, which I believe corresponds with the chassis numbering. On Cosmos, however, they are not:

[dogfood]root@oxz_switch0:~# pilot sp ls
CUBBY SERIAL      TYPE    IMAGE            IP
0     -           -       -                -
1     -           -       -                -
2     BRM22250001 cosmo   c6936cdddad8d869 fe80::aa40:25ff:fe04:c86
3     BRM13250012 cosmo   c6936cdddad8d869 fe80::aa40:25ff:fe04:402
4     -           -       -                -
5     -           -       -                -
6     -           -       -                -
7     BRM27230045 gimlet  aa915b62f157d678 fe80::aa40:25ff:fe04:6c6
8     BRM44220011 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:3c5
9     BRM44220005 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:347
10    BRM42220009 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:3c4
11    BRM42220006 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:181
12    BRM42220057 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:383
13    BRM42220018 gimlet  b8cf06f5c2c228cd fe80::aa40:25ff:fe04:1c1
14    BRM42220051 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:185
15    -           -       -                -
16    BRM42220014 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:342
17    BRM42220017 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:182
18    -           -       -                -
19    -           -       -                -
20    -           -       -                -
21    BRM42220031 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:343
22    -           -       -                -
23    BRM42220016 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:385
24    -           -       -                -
25    BRM44220010 gimlet  35c296036a3feb4b fe80::aa40:25ff:fe04:344
26    -           -       -                -
27    -           -       -                -
28    -           -       -                -
29    -           -       -                -
30    -           -       -                -
31    -           -       -                -

[dogfood]root@oxz_switch0:~# omdb db sleds
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (297.0.0)
note: listing all commissioned sleds (use -F to filter, e.g. -F in-service)
 SERIAL       IP                             ROLE      POLICY      STATE   BP AVAIL   ID
 BRM42220057  [fd00:1122:3344:104::1]:12345  -         in service  active  available  0c7011f7-a4bf-4daf-90cc-1c2410103300
 BRM44220011  [fd00:1122:3344:103::1]:12345  -         in service  active  available  2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa
 BRM22250001  [fd00:1122:3344:128::1]:12345  -         in service  active  available  4d0e70d5-a8ad-40b1-aa7b-209e2165bc75
 BRM42220017  [fd00:1122:3344:109::1]:12345  -         in service  active  available  5f6720b8-8a31-45f8-8c94-8e699218f28b
 BRM42220051  [fd00:1122:3344:10b::1]:12345  scrimlet  in service  active  available  71def415-55ad-46b4-ba88-3ca55d7fb287
 BRM13250012  [fd00:1122:3344:129::1]:12345  -         in service  active  available  781ab3c9-ff3c-428a-9f3e-895a1e86a111
 BRM27230045  [fd00:1122:3344:127::1]:12345  -         in service  active  available  7b473a3b-4ec2-4b58-8376-9b3cb68d1392
 BRM44220010  [fd00:1122:3344:101::1]:12345  -         in service  active  available  7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c
 BRM42220014  [fd00:1122:3344:108::1]:12345  scrimlet  in service  active  available  87c2c4fc-b0c7-4fef-a305-78f0ed265bbc
 BRM42220031  [fd00:1122:3344:102::1]:12345  -         in service  active  available  a2adea92-b56e-44fc-8a0d-7d63b5fd3b93
 BRM42220006  [fd00:1122:3344:106::1]:12345  -         in service  active  available  b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88
 BRM42220009  [fd00:1122:3344:107::1]:12345  -         in service  active  available  db183874-65b5-4263-a1c1-ddb2737ae0e9
 BRM42220016  [fd00:1122:3344:10a::1]:12345  -         in service  active  available  dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd
 BRM44220005  [fd00:1122:3344:105::1]:12345  -         in service  active  available  f15774c1-b8e5-434f-a493-ec43f96cba06

[dogfood]root@oxz_switch0:~# 

(note that 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 and 781ab3c9-ff3c-428a-9f3e-895a1e86a111 are the sled IDs of the two Cosmos, BRM22250001 and BRM13250012, respectively.)

[dogfood]root@oxz_switch0:~# omdb db inventory physical-disks
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (297.0.0)
INV_COLLECTION_ID                    SLED_ID                              SLOT VENDOR MODEL                     SERIAL       VARIANT FIRMWARE NEXT_FIRMWARE
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 0    1b96   WUS4C6432DSP3X3           A084A704     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 1    1b96   WUS4C6432DSP3X3           A084A5DA     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 2    1b96   WUS4C6432DSP3X3           A084A7DF     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 3    1b96   WUS4C6432DSP3X3           A084A5EE     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 4    1b96   WUS4C6432DSP3X3           A084A617     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 5    1b96   WUS4C6432DSP3X3           A084A6C9     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 6    1b96   WUS4C6432DSP3X3           A084A723     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 7    1b96   WUS4C6432DSP3X3           A084A643     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 8    1b96   WUS4C6432DSP3X3           A084A820     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 9    1b96   WUS4C6432DSP3X3           A084A60F     U2      R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 17   1344   Micron_7300_MTFDHBG1T9TDF 21413275374B M2      95420280
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 18   1344   Micron_7300_MTFDHBG1T9TDF 214132748192 M2      95420280

...

f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 32   1b96   WUS4C6432DSP3X3           A084A5EC     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 33   1b96   WUS4C6432DSP3X3           A084A7E5     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 34   1b96   WUS4C6432DSP3X3           A084A7CC     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 35   1b96   WUS4C6432DSP3X3           A084A68F     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 36   1b96   WUS4C6432DSP3X3           A084A73E     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 37   1b96   WUS4C6432DSP3X3           A084A77D     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 38   1b96   WUS4C6432DSP3X3           A084A5BE     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 39   1b96   WUS4C6432DSP3X3           A079E73D     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 40   1b96   WUS4C6432DSP3X3           A084A5F8     U2      R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 41   1b96   WUS4C6432DSP3X3           A079E2CD     U2      R2210010 R2210010

...

ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 17   1344   Micron_7450_MTFDKBG1T9TFR 22314564B5D8 M2      E2MU300
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 18   1344   Micron_7450_MTFDKBG1T9TFR 25034E674499 M2      E2MU300
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 32   1b96   WUS4C6432DSP3X3           A084A804     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 33   1b96   WUS4C6432DSP3X3           A079DF0A     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 34   1b96   WUS4C6432DSP3X3           A084A7E3     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 35   1b96   WUS4C6432DSP3X3           A079E760     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 36   1b96   WUS4C6432DSP3X3           A079E752     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 37   1b96   WUS4C6432DSP3X3           A079E7A5     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 38   1b96   WUS4C6432DSP3X3           A079E425     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 39   1b96   WUS4C6432DSP3X3           A079DFC5     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 40   1b96   WUS4C6432DSP3X3           A079E74E     U2      R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 41   1b96   WUS4C6432DSP3X3           A079E707     U2      R2210010 R2210010

...

[dogfood]root@oxz_switch0:~#

Here, we see a Gimlet with disks in slots 0-9, and the two Cosmos with disks numbered from 32-41.

Poking around a bit in the sled-agent code, it looks like this code in sled_hardware::illumos is where we actually determine the slot property to report:
https://github.com/oxidecomputer/omicron/blob/df990b0578fbee4afcf805423a20d85e23544e0d/sled-hardware/src/illumos/mod.rs#L450-L461

This is done by finding the pcieb driver node that's the parent of the nvme device for the disk, and then looking up the physical-slot# property on that pcieb node. I presume the reason we see different values here between Cosmo and Gimlet are due to differences between PCIe topology on the two boards.

In order to properly represent the location to the operator in the external API, we need a way to report the slot-number-as-marked-on-the-chassis in the inventory, as well. There are a few ways we could do this. For one, we could just have the API endpoints check if the sled is a Cosmo or Gimlet, and if it's a Cosmo, we could subtract 32 from the slot number before returning it in the API. But, this seems like a pretty disgusting hack.

Ideally, the sled and its host OS should remain the source of truth for these numbers. If possible, I'd much rather we find a way to ask the host OS for the information we actually want here, and include it in the inventory. This brings up a lot of questions which I don't currently know the answer to:

  1. Are the values currently returned for pcieb's physical-slot# property correct on Cosmo? I am not sure if this is intended behavior or a bug; if it's a bug, the solution here is, well, to fix the bug. If it's not, we would ideally find another way to get the information we want.
  2. If this is the intended behavior for pcieb physical-slot#, is there another way to get the slot-number-as-marked-on-the-chassis? Does the host OS know this information?
    a. If the host knows this, is there an existing API we can use? Perhaps we can do this using libtopo?
  3. Is important to also continue returning the slot numbers returned by the current code and including them in the inventory records in the database? Is something else using this and will something break if we replaced these values with the slot-number-as-marked-on-the-chassis? If it is, we will probably want to add any new information in a separate field/database column, so that we can also keep reporting the values from the pcieb node's physical-slot# property.
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/omicron

All issues in oxidecomputer/omicron

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.