oxidecomputer / oxidecomputer/hubris

Benign race condition in thermal loop

Open
#1,147 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

thermal 🚫🔥🚫
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

@bcantrill noticed that during a reboot loop, he occasionally saw a blip of high fan power:

humility: ring buffer task_thermal::__RINGBUF in thermal:
 NDX LINE      GEN    COUNT PAYLOAD
  10  884       74      119 ControlPwm(0x0)
  11  611       74        1 SensorReadFailed(SensorId(0x35), SensorFailure)
  12  611       74        1 SensorReadFailed(SensorId(0x36), SensorFailure)
  13  884       74       49 ControlPwm(0x0)
  14  678       74        1 PowerModeChanged(PowerBitmask { bits: 0x1 })
  15  555       74        1 AutoState(Boot)
  16  770       74        1 AutoState(Running)
  17  884       74        6 ControlPwm(0x0)
  18  678       74        1 PowerModeChanged(PowerBitmask { bits: 0x2 })
  19  555       74        1 AutoState(Boot)
  20  770       74        1 AutoState(Running)
  21  884       74        1 ControlPwm(0x0)
  22  678       74        1 PowerModeChanged(PowerBitmask { bits: 0xe })
  23  555       74        1 AutoState(Boot)
  24  770       74        1 AutoState(Running)
  25  884       74      119 ControlPwm(0x0)
  26  611       74        1 SensorReadFailed(SensorId(0x35), SensorFailure)
  27  611       74        1 SensorReadFailed(SensorId(0x36), SensorFailure)
  28  884       74       47 ControlPwm(0x0)
  29  678       74        1 PowerModeChanged(PowerBitmask { bits: 0x1 })
  30  555       74        1 AutoState(Boot)
  31  770       74        1 AutoState(Running)
   0  884       75        5 ControlPwm(0x0)
   1  678       75        1 PowerModeChanged(PowerBitmask { bits: 0x2 })
   2  555       75        1 AutoState(Boot)
   3  770       75        1 AutoState(Running)
   4  884       75        1 ControlPwm(0x0)
   5  678       75        1 PowerModeChanged(PowerBitmask { bits: 0xe })
   6  555       75        1 AutoState(Boot)
   7  884       75        1 ControlPwm(0x64)
   8  770       75        1 AutoState(Running)
   9  884       75       57 ControlPwm(0x0)

(note that NDX = 7 has 100% fan power, for one iteration of the thermal loop)

This is due to the recent decoupling of thermal and sensors (#1052), and is a benign race condition:

  • We check the power mode to decide which sensors to poll, poll them, and send data to the sensors task
  • The system is powered on at exactly this moment!
  • We check the power mode to decide which sensors to use as inputs to the thermal loop, and read them from sensors
  • There are new sensors with unknown values, so we stay in Boot (at 100% fan power) for one iteration through the thermal loop
  • Next time through the loop, we poll the correct set of sensors, so they've all got data and we transition to Running

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the thermal and sensors tasks described in the issue, including the power-mode checks and the handoff introduced by #1052. The issue does not name files, tests, or a specific corrective change, so the desired resolution and completion criteria need to be clarified before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.