oxidecomputer / oxidecomputer/hubris
Benign race condition in thermal loop
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
@bcantrill noticed that during a reboot loop, he occasionally saw a blip of high fan power:
humility: ring buffer task_thermal::__RINGBUF in thermal:
NDX LINE GEN COUNT PAYLOAD
10 884 74 119 ControlPwm(0x0)
11 611 74 1 SensorReadFailed(SensorId(0x35), SensorFailure)
12 611 74 1 SensorReadFailed(SensorId(0x36), SensorFailure)
13 884 74 49 ControlPwm(0x0)
14 678 74 1 PowerModeChanged(PowerBitmask { bits: 0x1 })
15 555 74 1 AutoState(Boot)
16 770 74 1 AutoState(Running)
17 884 74 6 ControlPwm(0x0)
18 678 74 1 PowerModeChanged(PowerBitmask { bits: 0x2 })
19 555 74 1 AutoState(Boot)
20 770 74 1 AutoState(Running)
21 884 74 1 ControlPwm(0x0)
22 678 74 1 PowerModeChanged(PowerBitmask { bits: 0xe })
23 555 74 1 AutoState(Boot)
24 770 74 1 AutoState(Running)
25 884 74 119 ControlPwm(0x0)
26 611 74 1 SensorReadFailed(SensorId(0x35), SensorFailure)
27 611 74 1 SensorReadFailed(SensorId(0x36), SensorFailure)
28 884 74 47 ControlPwm(0x0)
29 678 74 1 PowerModeChanged(PowerBitmask { bits: 0x1 })
30 555 74 1 AutoState(Boot)
31 770 74 1 AutoState(Running)
0 884 75 5 ControlPwm(0x0)
1 678 75 1 PowerModeChanged(PowerBitmask { bits: 0x2 })
2 555 75 1 AutoState(Boot)
3 770 75 1 AutoState(Running)
4 884 75 1 ControlPwm(0x0)
5 678 75 1 PowerModeChanged(PowerBitmask { bits: 0xe })
6 555 75 1 AutoState(Boot)
7 884 75 1 ControlPwm(0x64)
8 770 75 1 AutoState(Running)
9 884 75 57 ControlPwm(0x0)
(note that NDX = 7 has 100% fan power, for one iteration of the thermal loop)
This is due to the recent decoupling of thermal and sensors (#1052), and is a benign race condition:
- We check the power mode to decide which sensors to poll, poll them, and send data to the
sensorstask - The system is powered on at exactly this moment!
- We check the power mode to decide which sensors to use as inputs to the thermal loop, and read them from
sensors - There are new sensors with unknown values, so we stay in
Boot(at 100% fan power) for one iteration through the thermal loop - Next time through the loop, we poll the correct set of sensors, so they've all got data and we transition to
Running
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the thermal and sensors tasks described in the issue, including the power-mode checks and the handoff introduced by #1052. The issue does not name files, tests, or a specific corrective change, so the desired resolution and completion criteria need to be clarified before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100