sled-agent `physical_disk` inventory should include topo's `label`, rather than `pcieb`'s `physical-slot#`
@smklein is already working on this.
Since Sep 9, 2026.
Assessment
This issue has not been assessed yet.
Description
See https://github.com/oxidecomputer/omicron/pull/11221#issuecomment-5574308943, https://github.com/oxidecomputer/omicron/pull/11221#issuecomment-5574551780, and https://github.com/oxidecomputer/omicron/pull/11221#pullrequestreview-5134670273.
As described in https://github.com/oxidecomputer/omicron/issues/11088, we would like to be able to include a slot number in the external API's physical_disk list and views. The value we return here needs to be the slot number marked on the chassis, since the intent behind such an API would be to allow an operator to identify where a disk is located within a sled for purposes such as disk replacement. Naively, I had written in #11088 that we could do this by using the inventory to look up a disk's sled by ID and then look up the slot in the inventory entry for that disk.
On Gimlets, the values for the slot field in the inventory's physical disk records are numbered 0-9 for U.2s, which I believe corresponds with the chassis numbering. On Cosmos, however, they are not:
[dogfood]root@oxz_switch0:~# pilot sp ls
CUBBY SERIAL TYPE IMAGE IP
0 - - - -
1 - - - -
2 BRM22250001 cosmo c6936cdddad8d869 fe80::aa40:25ff:fe04:c86
3 BRM13250012 cosmo c6936cdddad8d869 fe80::aa40:25ff:fe04:402
4 - - - -
5 - - - -
6 - - - -
7 BRM27230045 gimlet aa915b62f157d678 fe80::aa40:25ff:fe04:6c6
8 BRM44220011 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:3c5
9 BRM44220005 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:347
10 BRM42220009 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:3c4
11 BRM42220006 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:181
12 BRM42220057 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:383
13 BRM42220018 gimlet b8cf06f5c2c228cd fe80::aa40:25ff:fe04:1c1
14 BRM42220051 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:185
15 - - - -
16 BRM42220014 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:342
17 BRM42220017 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:182
18 - - - -
19 - - - -
20 - - - -
21 BRM42220031 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:343
22 - - - -
23 BRM42220016 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:385
24 - - - -
25 BRM44220010 gimlet 35c296036a3feb4b fe80::aa40:25ff:fe04:344
26 - - - -
27 - - - -
28 - - - -
29 - - - -
30 - - - -
31 - - - -
[dogfood]root@oxz_switch0:~# omdb db sleds
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (297.0.0)
note: listing all commissioned sleds (use -F to filter, e.g. -F in-service)
SERIAL IP ROLE POLICY STATE BP AVAIL ID
BRM42220057 [fd00:1122:3344:104::1]:12345 - in service active available 0c7011f7-a4bf-4daf-90cc-1c2410103300
BRM44220011 [fd00:1122:3344:103::1]:12345 - in service active available 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa
BRM22250001 [fd00:1122:3344:128::1]:12345 - in service active available 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75
BRM42220017 [fd00:1122:3344:109::1]:12345 - in service active available 5f6720b8-8a31-45f8-8c94-8e699218f28b
BRM42220051 [fd00:1122:3344:10b::1]:12345 scrimlet in service active available 71def415-55ad-46b4-ba88-3ca55d7fb287
BRM13250012 [fd00:1122:3344:129::1]:12345 - in service active available 781ab3c9-ff3c-428a-9f3e-895a1e86a111
BRM27230045 [fd00:1122:3344:127::1]:12345 - in service active available 7b473a3b-4ec2-4b58-8376-9b3cb68d1392
BRM44220010 [fd00:1122:3344:101::1]:12345 - in service active available 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c
BRM42220014 [fd00:1122:3344:108::1]:12345 scrimlet in service active available 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc
BRM42220031 [fd00:1122:3344:102::1]:12345 - in service active available a2adea92-b56e-44fc-8a0d-7d63b5fd3b93
BRM42220006 [fd00:1122:3344:106::1]:12345 - in service active available b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88
BRM42220009 [fd00:1122:3344:107::1]:12345 - in service active available db183874-65b5-4263-a1c1-ddb2737ae0e9
BRM42220016 [fd00:1122:3344:10a::1]:12345 - in service active available dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd
BRM44220005 [fd00:1122:3344:105::1]:12345 - in service active available f15774c1-b8e5-434f-a493-ec43f96cba06
[dogfood]root@oxz_switch0:~#
(note that 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 and 781ab3c9-ff3c-428a-9f3e-895a1e86a111 are the sled IDs of the two Cosmos, BRM22250001 and BRM13250012, respectively.)
[dogfood]root@oxz_switch0:~# omdb db inventory physical-disks
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (297.0.0)
INV_COLLECTION_ID SLED_ID SLOT VENDOR MODEL SERIAL VARIANT FIRMWARE NEXT_FIRMWARE
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 0 1b96 WUS4C6432DSP3X3 A084A704 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 1 1b96 WUS4C6432DSP3X3 A084A5DA U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 2 1b96 WUS4C6432DSP3X3 A084A7DF U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 3 1b96 WUS4C6432DSP3X3 A084A5EE U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 4 1b96 WUS4C6432DSP3X3 A084A617 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 5 1b96 WUS4C6432DSP3X3 A084A6C9 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 6 1b96 WUS4C6432DSP3X3 A084A723 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 7 1b96 WUS4C6432DSP3X3 A084A643 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 8 1b96 WUS4C6432DSP3X3 A084A820 U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 9 1b96 WUS4C6432DSP3X3 A084A60F U2 R2210010 R2210010
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 17 1344 Micron_7300_MTFDHBG1T9TDF 21413275374B M2 95420280
1532212c-ef05-4b64-a25e-2551a22176a5 0c7011f7-a4bf-4daf-90cc-1c2410103300 18 1344 Micron_7300_MTFDHBG1T9TDF 214132748192 M2 95420280
...
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 32 1b96 WUS4C6432DSP3X3 A084A5EC U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 33 1b96 WUS4C6432DSP3X3 A084A7E5 U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 34 1b96 WUS4C6432DSP3X3 A084A7CC U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 35 1b96 WUS4C6432DSP3X3 A084A68F U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 36 1b96 WUS4C6432DSP3X3 A084A73E U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 37 1b96 WUS4C6432DSP3X3 A084A77D U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 38 1b96 WUS4C6432DSP3X3 A084A5BE U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 39 1b96 WUS4C6432DSP3X3 A079E73D U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 40 1b96 WUS4C6432DSP3X3 A084A5F8 U2 R2210010 R2210010
f6c0c0ab-1f8e-49ae-b009-4b24e1c504b3 4d0e70d5-a8ad-40b1-aa7b-209e2165bc75 41 1b96 WUS4C6432DSP3X3 A079E2CD U2 R2210010 R2210010
...
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 17 1344 Micron_7450_MTFDKBG1T9TFR 22314564B5D8 M2 E2MU300
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 18 1344 Micron_7450_MTFDKBG1T9TFR 25034E674499 M2 E2MU300
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 32 1b96 WUS4C6432DSP3X3 A084A804 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 33 1b96 WUS4C6432DSP3X3 A079DF0A U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 34 1b96 WUS4C6432DSP3X3 A084A7E3 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 35 1b96 WUS4C6432DSP3X3 A079E760 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 36 1b96 WUS4C6432DSP3X3 A079E752 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 37 1b96 WUS4C6432DSP3X3 A079E7A5 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 38 1b96 WUS4C6432DSP3X3 A079E425 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 39 1b96 WUS4C6432DSP3X3 A079DFC5 U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 40 1b96 WUS4C6432DSP3X3 A079E74E U2 R2210010 R2210010
ab34fa8b-482b-4eea-a928-3c547b4a9002 781ab3c9-ff3c-428a-9f3e-895a1e86a111 41 1b96 WUS4C6432DSP3X3 A079E707 U2 R2210010 R2210010
...
[dogfood]root@oxz_switch0:~#
Here, we see a Gimlet with disks in slots 0-9, and the two Cosmos with disks numbered from 32-41.
Poking around a bit in the sled-agent code, it looks like this code in sled_hardware::illumos is where we actually determine the slot property to report:
https://github.com/oxidecomputer/omicron/blob/df990b0578fbee4afcf805423a20d85e23544e0d/sled-hardware/src/illumos/mod.rs#L450-L461
This is done by finding the pcieb driver node that's the parent of the nvme device for the disk, and then looking up the physical-slot# property on that pcieb node. I presume the reason we see different values here between Cosmo and Gimlet are due to differences between PCIe topology on the two boards.
In order to properly represent the location to the operator in the external API, we need a way to report the slot-number-as-marked-on-the-chassis in the inventory, as well. There are a few ways we could do this. For one, we could just have the API endpoints check if the sled is a Cosmo or Gimlet, and if it's a Cosmo, we could subtract 32 from the slot number before returning it in the API. But, this seems like a pretty disgusting hack.
Ideally, the sled and its host OS should remain the source of truth for these numbers. If possible, I'd much rather we find a way to ask the host OS for the information we actually want here, and include it in the inventory. This brings up a lot of questions which I don't currently know the answer to:
- Are the values currently returned for
pcieb'sphysical-slot#property correct on Cosmo? I am not sure if this is intended behavior or a bug; if it's a bug, the solution here is, well, to fix the bug. If it's not, we would ideally find another way to get the information we want. - If this is the intended behavior for
pciebphysical-slot#, is there another way to get the slot-number-as-marked-on-the-chassis? Does the host OS know this information?
a. If the host knows this, is there an existing API we can use? Perhaps we can do this usinglibtopo? - Is important to also continue returning the slot numbers returned by the current code and including them in the inventory records in the database? Is something else using this and will something break if we replaced these values with the slot-number-as-marked-on-the-chassis? If it is, we will probably want to add any new information in a separate field/database column, so that we can also keep reporting the values from the
pciebnode'sphysical-slot#property.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from oxidecomputer/omicron
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
oxidecomputer/omicron#11269 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
oxidecomputer/omicron#11266 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
oxidecomputer/omicron#11260 · 1 comment ·
-
wicket's errors should be better when trying to read sensitive data from ssh without a pseudo-tty Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
oxidecomputer/omicron#11148 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
oxidecomputer/omicron#10907 ·
All issues in oxidecomputer/omicron
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
kwakseongjae/auto-hwp#319 ·
-
area:cli bug filter-quality good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
bevyengine/bevy#25861 ·
-
comp-datalake
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
ClickHouse/ClickHouse#121222 ·
-
enhancement remote
Difficulty 2/5 1-3 hours Newbie friendliness 68/100