oxidecomputer / oxidecomputer/hubris
non-I2C sensors don't make it into control-plane-agent inventory and are thus not recorded by MGS
Open
Nobody has claimed this yet.
control-plane-agent
cosmo
fault-management
service processor
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
While working on mostly-unrelated changes to sensors codegen, I discovered something quite unfortunate. Any sensors which are declared via the Sensors task's config ([[config.sensor.devices]]) rather than through the I2C driver config ([[config.i2c.devices]] with a sensors = ... key in it) are not present in the control-plane-agent inventory and component-details APIs.
They can still be read through the control-plane-agent API's read-sensor-value request with the sensor ID for the device, but this is not presently used by the control plane, as it requires knowledge of which sensor IDs correspond to which devices, which the control plane lacks. Instead, the control plane uses inventory to discover the SP's components and then requests sensor data by calling component-details for any devices which advertise measurement channels. This means that any sensors not implemented using the I2C task's codegen are not being polled by the control plane and recorded in ClickHouse unless the device is manually special-cased in control-plane-agent, which most of them are not.
In particular, devices which are implemented using [[config.sensor.devices]] and are thus not present in the control-plane-agent inventory include DIMM temperature sensors on Cosmo and transceiver temperatures on Sidecar.
For example, on Gimlet, DIMM temperature sensors are declared as I2C devices through the I2C driver config:
https://github.com/oxidecomputer/hubris/blob/9452975dd8953b6bb8f219bb131e7dfa59fa65ae/app/gimlet/base.toml#L1075-L1234
Asking a Gimlet for its inventory includes the DIMMs and their temperature sensors:
eliza@castle ~ $ pilot sp exec -e 'inventory' BRM42220031
Feb 09 20:57:09.157 INFO creating SP handle on interface rack2sw0tp0, component: faux-mgs
Feb 09 20:57:09.161 INFO initial discovery complete, addr: [fe80::aa40:25ff:fe04:343%13]:11111, interface: rack2sw0tp0, socket: control-plane-agent, component: faux-mgs
COMPONENT STATUS DEVICE DESCRIPTION (CAPABILITIES)
sp Present sp Service Processor (DeviceCapabilities(1))
sp3-host-cpu Present sp3-host-cpu Gimlet SP3 host cpu (DeviceCapabilities(4))
host-boot-flash Present host-boot-flash Gimlet host boot flash (DeviceCapabilities(1))
system-led Present system-led System attention LED (DeviceCapabilities(8))
J194/U1 Present tmp117 Southwest temperature sensor (DeviceCapabilities(2))
J195/U1 Present tmp117 South temperature sensor (DeviceCapabilities(2))
J196/U1 Present tmp117 Southeast temperature sensor (DeviceCapabilities(2))
U336 Present pca9545 U.2 ABCD mux (DeviceCapabilities(0))
U339 Present pca9545 U.2 EFGH mux (DeviceCapabilities(0))
U337 Present pca9545 U.2 IJ/FRUID mux (DeviceCapabilities(0))
J206/U7 Present at24csw080 U.2 Sharkfin A VPD (DeviceCapabilities(0))
J206/U8 Present max5970 U.2 Sharkfin A hot swap controller (DeviceCapabilities(2))
J206/J1 Present nvme_bmc U.2 A NVMe Basic Management Command (DeviceCapabilities(2))
J207/U7 Present at24csw080 U.2 Sharkfin B VPD (DeviceCapabilities(0))
J207/U8 Present max5970 U.2 Sharkfin B hot swap controller (DeviceCapabilities(2))
J207/J1 Present nvme_bmc U.2 B NVMe Basic Management Control (DeviceCapabilities(2))
J208/U7 Present at24csw080 U.2 Sharkfin C VPD (DeviceCapabilities(0))
J208/U8 Present max5970 U.2 Sharkfin C hot swap controller (DeviceCapabilities(2))
J208/J1 Present nvme_bmc U.2 C NVMe Basic Management Control (DeviceCapabilities(2))
J209/U7 Present at24csw080 U.2 Sharkfin D VPD (DeviceCapabilities(0))
J209/U8 Present max5970 U.2 Sharkfin D hot swap controller (DeviceCapabilities(2))
J209/J1 Present nvme_bmc U.2 D NVMe Basic Management Control (DeviceCapabilities(2))
J210/U7 Present at24csw080 U.2 Sharkfin E VPD (DeviceCapabilities(0))
J210/U8 Present max5970 U.2 Sharkfin E hot swap controller (DeviceCapabilities(2))
J210/J1 Present nvme_bmc U.2 E NVMe Basic Management Control (DeviceCapabilities(2))
J211/U7 Present at24csw080 U.2 Sharkfin F VPD (DeviceCapabilities(0))
J211/U8 Present max5970 U.2 Sharkfin F hot swap controller (DeviceCapabilities(2))
J211/J1 Present nvme_bmc U.2 F NVMe Basic Management Control (DeviceCapabilities(2))
J212/U7 Present at24csw080 U.2 Sharkfin G VPD (DeviceCapabilities(0))
J212/U8 Present max5970 U.2 Sharkfin G hot swap controller (DeviceCapabilities(2))
J212/J1 Present nvme_bmc U.2 G NVMe Basic Management Control (DeviceCapabilities(2))
J213/U7 Present at24csw080 U.2 Sharkfin H VPD (DeviceCapabilities(0))
J213/U8 Present max5970 U.2 Sharkfin H hot swap controller (DeviceCapabilities(2))
J213/J1 Present nvme_bmc U.2 H NVMe Basic Management Control (DeviceCapabilities(2))
J214/U7 Present at24csw080 U.2 Sharkfin I VPD (DeviceCapabilities(0))
J214/U8 Present max5970 U.2 Sharkfin I hot swap controller (DeviceCapabilities(2))
J214/J1 Present nvme_bmc U.2 I NVMe Basic Management Control (DeviceCapabilities(2))
J215/U7 Present at24csw080 U.2 Sharkfin J VPD (DeviceCapabilities(0))
J215/U8 Present max5970 U.2 Sharkfin J hot swap controller (DeviceCapabilities(2))
J215/J1 Present nvme_bmc U.2 J NVMe Basic Management Control (DeviceCapabilities(2))
U615 Present at24csw080 Gimlet VPD (DeviceCapabilities(0))
U422 Present pca9545 M.2 mux (DeviceCapabilities(0))
J180/U1 Present at24csw080 Fan VPD (DeviceCapabilities(0))
U491 Present tmp451 T6 temperature sensor (DeviceCapabilities(2))
U522 Present tps546b24a A2 3.3V rail (DeviceCapabilities(2))
U560 Present tps546b24a A0 3.3V rail (DeviceCapabilities(2))
U524 Present tps546b24a A2 5V rail (DeviceCapabilities(2))
U561 Present tps546b24a A2 1.8V rail (DeviceCapabilities(2))
U275 Present max5970 M.2 hot plug controller (DeviceCapabilities(2))
P0/SBRMI Present sbrmi CPU via SB-RMI (DeviceCapabilities(0))
P0/SBTSI Present sbtsi CPU temperature sensor (DeviceCapabilities(2))
U446 Present idt8a34003 Clock generator (DeviceCapabilities(0))
U350 Present raa229618 CPU power controller (DeviceCapabilities(2))
U351 Present raa229618 SoC power controller (DeviceCapabilities(2))
U352 Present isl68224 DIMM/SP3 1.8V A0 power controller (DeviceCapabilities(2))
U419 Present adm127x Fan hot swap controller (DeviceCapabilities(2))
U452 Present adm127x Sled hot swap controller (DeviceCapabilities(2))
U321 Present max31790 Fan controller (DeviceCapabilities(2))
U565 Present tps546b24a T6 power controller (DeviceCapabilities(2))
J197/U1 Present tmp117 Northeast temperature sensor (DeviceCapabilities(2))
J198/U1 Present tmp117 North temperature sensor (DeviceCapabilities(2))
J199/U1 Present tmp117 Northwest temperature sensor (DeviceCapabilities(2))
U431 Present bmr491 Intermediate bus converter (DeviceCapabilities(2))
M0 Present tse2004av DIMM A0 (DeviceCapabilities(2))
M8 Present tse2004av DIMM A1 (DeviceCapabilities(2))
M1 Present tse2004av DIMM B0 (DeviceCapabilities(2))
M9 Present tse2004av DIMM B1 (DeviceCapabilities(2))
M2 Present tse2004av DIMM C0 (DeviceCapabilities(2))
M10 Present tse2004av DIMM C1 (DeviceCapabilities(2))
M3 Present tse2004av DIMM D0 (DeviceCapabilities(2))
M11 Present tse2004av DIMM D1 (DeviceCapabilities(2))
M4 Present tse2004av DIMM E0 (DeviceCapabilities(2))
M12 Present tse2004av DIMM E1 (DeviceCapabilities(2))
M5 Present tse2004av DIMM F0 (DeviceCapabilities(2))
M13 Present tse2004av DIMM F1 (DeviceCapabilities(2))
M6 Present tse2004av DIMM G0 (DeviceCapabilities(2))
M14 Present tse2004av DIMM G1 (DeviceCapabilities(2))
M7 Present tse2004av DIMM H0 (DeviceCapabilities(2))
M15 Present tse2004av DIMM H1 (DeviceCapabilities(2))
P1 Unavailable m2_hp_only M.2 A NVMe Basic Management Command (DeviceCapabilities(2))
P2 Unavailable m2_hp_only M.2 B NVMe Basic Management Command (DeviceCapabilities(2))
eliza@castle ~ $
Meanwhile, on Cosmo, the DIMM temperature sensors are declared here, as their I2C interfaces are proxied through the FPGA:
https://github.com/oxidecomputer/hubris/blob/9452975dd8953b6bb8f219bb131e7dfa59fa65ae/app/cosmo/base.toml#L1234-L1377
When I ask a Cosmo for its inventory, I get this:
eliza@castle ~ $ pilot sp exec -e 'inventory' BRM22250001
Feb 09 20:55:35.930 INFO creating SP handle on interface rack2sw0tp0, component: faux-mgs
Feb 09 20:55:35.935 INFO initial discovery complete, addr: [fe80::aa40:25ff:fe04:c86%13]:11111, interface: rack2sw0tp0, socket: control-plane-agent, component: faux-mgs
COMPONENT STATUS DEVICE DESCRIPTION (CAPABILITIES)
sp Present sp Service Processor (DeviceCapabilities(1))
sp-aux-flash Present sp-aux-flash Service Processor auxiliary flash (DeviceCapabilities(0))
sp5-host-cpu Present sp5-host-cpu Cosmo SP5 host cpu (DeviceCapabilities(4))
host-boot-flash Present host-boot-flash Cosmo host boot flash (DeviceCapabilities(1))
system-led Present system-led System attention LED (DeviceCapabilities(8))
U31 Present oximux16 Front FPGA virtual mux (DeviceCapabilities(0))
J200/U2 Present at24csw080 U.2 Sharkfin A VPD (DeviceCapabilities(0))
J200/U1 Present max5970 U.2 Sharkfin A hot swap controller (DeviceCapabilities(2))
J200/J2 Present nvme_bmc U.2 A NVMe Basic Management Command (DeviceCapabilities(2))
J201/U2 Present at24csw080 U.2 Sharkfin B VPD (DeviceCapabilities(0))
J201/U1 Present max5970 U.2 Sharkfin B hot swap controller (DeviceCapabilities(2))
J201/J2 Present nvme_bmc U.2 B NVMe Basic Management Control (DeviceCapabilities(2))
J202/U2 Present at24csw080 U.2 Sharkfin C VPD (DeviceCapabilities(0))
J202/U1 Present max5970 U.2 Sharkfin C hot swap controller (DeviceCapabilities(2))
J202/J2 Present nvme_bmc U.2 C NVMe Basic Management Control (DeviceCapabilities(2))
J203/U2 Present at24csw080 U.2 Sharkfin D VPD (DeviceCapabilities(0))
J203/U1 Present max5970 U.2 Sharkfin D hot swap controller (DeviceCapabilities(2))
J203/J2 Present nvme_bmc U.2 D NVMe Basic Management Control (DeviceCapabilities(2))
J204/U2 Present at24csw080 U.2 Sharkfin E VPD (DeviceCapabilities(0))
J204/U1 Present max5970 U.2 Sharkfin E hot swap controller (DeviceCapabilities(2))
J204/J2 Present nvme_bmc U.2 E NVMe Basic Management Control (DeviceCapabilities(2))
J205/U2 Present at24csw080 U.2 Sharkfin F VPD (DeviceCapabilities(0))
J205/U1 Present max5970 U.2 Sharkfin F hot swap controller (DeviceCapabilities(2))
J205/J2 Present nvme_bmc U.2 F NVMe Basic Management Control (DeviceCapabilities(2))
J206/U2 Present at24csw080 U.2 Sharkfin G VPD (DeviceCapabilities(0))
J206/U1 Present max5970 U.2 Sharkfin G hot swap controller (DeviceCapabilities(2))
J206/J2 Present nvme_bmc U.2 G NVMe Basic Management Control (DeviceCapabilities(2))
J207/U2 Present at24csw080 U.2 Sharkfin H VPD (DeviceCapabilities(0))
J207/U1 Present max5970 U.2 Sharkfin H hot swap controller (DeviceCapabilities(2))
J207/J2 Present nvme_bmc U.2 H NVMe Basic Management Control (DeviceCapabilities(2))
J208/U2 Present at24csw080 U.2 Sharkfin I VPD (DeviceCapabilities(0))
J208/U1 Present max5970 U.2 Sharkfin I hot swap controller (DeviceCapabilities(2))
J208/J2 Present nvme_bmc U.2 I NVMe Basic Management Control (DeviceCapabilities(2))
J209/U2 Present at24csw080 U.2 Sharkfin J VPD (DeviceCapabilities(0))
J209/U1 Present max5970 U.2 Sharkfin J hot swap controller (DeviceCapabilities(2))
J209/J2 Present nvme_bmc U.2 J NVMe Basic Management Control (DeviceCapabilities(2))
J44/U1 Present tmp117 Southwest temperature sensor (DeviceCapabilities(2))
J45/U1 Present tmp117 South temperature sensor (DeviceCapabilities(2))
J46/U1 Present tmp117 Southeast temperature sensor (DeviceCapabilities(2))
U27/MUX Present oximux16 Main FPGA virtual mux (DeviceCapabilities(0))
J210 Present nvme_bmc M.2 A NVMe Basic Management Command (DeviceCapabilities(2))
J211 Present nvme_bmc M.2 B NVMe Basic Management Command (DeviceCapabilities(2))
U1/SBRMI Present sbrmi CPU via SB-RMI (DeviceCapabilities(0))
U1/SBTSI Present sbtsi CPU temperature sensor (DeviceCapabilities(2))
J34/U1 Present at24csw080 Fan VPD (DeviceCapabilities(0))
U53 Present tmp451 T6 temperature sensor (DeviceCapabilities(2))
U82 Present tps546b24a A2 3.3V rail (DeviceCapabilities(2))
U83 Present tps546b24a A2 5V rail (DeviceCapabilities(2))
U81 Present tps546b24a A2 1.8V rail (DeviceCapabilities(2))
U15 Present max5970 M.2 hot plug controller (DeviceCapabilities(2))
U16 Present ltc4282 12V MCIO hot plug controller (DeviceCapabilities(2))
U42 Present ltc4282 DIMM GHIJKL hot plug controller (DeviceCapabilities(2))
U127 Present ltc4282 DIMM ABCDEF hot plug controller (DeviceCapabilities(2))
U90 Present raa229620a South power controller (Core 0, SOC) (DeviceCapabilities(2))
U103 Present raa229620a North power controller (Core 1, VDDIO) (DeviceCapabilities(2))
U116 Present isl68224 SP5 power controller (V1P1, V1P8, V3P3) (DeviceCapabilities(2))
U54 Present max5970 NIC hot swap (DeviceCapabilities(2))
U123 Present tps546b24a T6 power controller (DeviceCapabilities(2))
J47/U1 Present tmp117 Northwest temperature sensor (DeviceCapabilities(2))
J48/U1 Present tmp117 North temperature sensor (DeviceCapabilities(2))
J49/U1 Present tmp117 Northeast temperature sensor (DeviceCapabilities(2))
U58 Present max31790 Fan controller (DeviceCapabilities(2))
U80 Present bmr491 Intermediate bus converter (DeviceCapabilities(2))
U32 Present at24csw080 Cosmo VPD (DeviceCapabilities(0))
U71 Present lm5066i Fan hot swap controller (east) (DeviceCapabilities(2))
U72 Present lm5066i Fan hot swap controller (central) (DeviceCapabilities(2))
U73 Present lm5066i Fan hot swap controller (west) (DeviceCapabilities(2))
U79 Present adm127x Sled hot swap controller (DeviceCapabilities(2))
eliza@castle ~ $
Note the distinct lack of DIMMs and their temperatures.
Similarly, on Sidecar, QSFP management interface sensors are declared through the sensor task, as their I2C interfaces are also proxied through an FPGA:
Therefore, the Sidecar's inventory output similarly doesn't know about transceivers. This is quite unfortunate!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the control-plane-agent inventory and component-details handling, then read the gateway/src/metrics.rs references showing how components and measurement channels are discovered. Compare those paths with the sensor declarations in app/gimlet/base.toml and app/cosmo/base.toml. Done means non-I2C sensors such as DIMM and transceiver temperature sensors appear in inventory and are available to control-plane polling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100