Cosmo's thermal loop does not consider DIMM temperatures

Open
#2,612 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
rust
Domain
embedded-iot

Research direction

Start with task/thermal/src/bsp/cosmo_ab.rs and the existing DIMM temperature handling in task/cosmo-spd/src/main.rs. Read the linked historical issues and pull requests before choosing between thermal inputs and dynamic_inputs, including the required polling and IPC behavior. Done means Cosmo’s thermal loop safely accounts for the available DIMM sensor data without interfering with SPD polling.

Written by the indexing model from the issue text.

Description

cosmo thermal 🚫🔥🚫

There is still a todo in cosmo's thermal BSP that notes that we do not consider the DIMM temperatures as part of our control loop.

There are a few historical reasons why this is the case, including some limitations in earlier hardware revisions that required a bit more care when polling DIMM temperatures on the SPD bus:

However, these days, cosmo-spd does measure these temperature sensors at appropriate times, and posts them to the sensor task already:

https://github.com/oxidecomputer/hubris/blob/f465a976476e28aae6c8adbf4b8b2321573088b0/task/cosmo-spd/src/main.rs#L289-L290

However, this data is not currently considered by Cosmo's thermal loop.

There are two ways we could resolve this:

  1. Add the DIMM sensor definitions to the inputs of cosmo, for the thermal loop to poll the sensors directly over I2C
  2. Add the DIMM sensor definitions to the dynamic_inputs of cosmo, for the thermal loop to poll for data from the sensor task

For solution 1, we would need to ensure that we don't interfere with the work that the spd task is doing, and that we only poll at appropriate times.

For solution 2, we would need cosmo-spd to also send IPC commands to thermal to enable/disable the dynamic inputs when entering/exiting states where the DIMM temperature data would be obtained. One challenge here is that SPD has a lower priority number (more important) than thermal, which means it cannot directly send IPC requests to the higher priority number (less important) thermal task. We could implement a pingback pattern for this though.

Chatting with @nathanaelhuffman, we believe that it's unlikely that the DIMMs would be overtemp when the CPU is not, though it would likely be good to eventually resolve this.

As the data is available in the sensors task, we may consider observing this in-situ using the MGS link to observe the sensor values, if we can find a suitable system we can use for testing, to ensure that the DIMMs are not operating hotter than we expect. This would help guide priority decisions for this.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/hubris

All issues in oxidecomputer/hubris

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.