oxidecomputer / oxidecomputer/hubris

recent spate of cosmos going full-fan associated with I2C `CorruptReply` errors

Open
#2,643 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

thermal 🚫🔥🚫
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

Recently, we have seen some otherwise idle Cosmos in the office start running fans on full blast. While we don't have a ringbuf capture showing the actual period of time during which we were on full-fan, we have seen a lot of CorruptReply I2C errors in the thermal ringbuf. For instance, on today's fan event on cosmo-os, I see:

eliza@castle ~ $ humility -d /staff/luqman/fans-go/hubris.core.0 ringbuf thermal
humility: attached to dump
humility: ring buffer drv_i2c_devices::emc2305::__RINGBUF in thermal:
humility: ring buffer drv_i2c_devices::max31790::__RINGBUF in thermal:
humility: ring buffer task_thermal::__RINGBUF in thermal:
   TOTAL VARIANT
 1381043 ControlPwm
     192 SensorReadFailed
      56 AutoState(Boot)
      55 AutoState(Running)
      55 PowerModeChanged
      44 MiscReadFailed
       6 FanAdded
       1 Start
       1 ThermalMode(Auto)
       1 FanControllerInitialized
       1 SetFanWatchdogOk
 NDX LINE      GEN    COUNT PAYLOAD
  27 1231       58        1 SensorReadFailed(SensorId(0x9), CorruptReply)
  28 1496       58       25 ControlPwm(0x0)
  29 1231       58        1 SensorReadFailed(SensorId(0x13), CorruptReply)
  30 1231       58        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  31 1496       58        1 ControlPwm(0x0)
   0 1231       59        1 SensorReadFailed(SensorId(0x13), CorruptReply)
   1 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
   2 1496       59       11 ControlPwm(0x0)
   3 1231       59        1 SensorReadFailed(SensorId(0x27), CorruptReply)
   4 1496       59        1 ControlPwm(0x0)
   5 1231       59        1 SensorReadFailed(SensorId(0x27), CorruptReply)
   6 1496       59       29 ControlPwm(0x0)
   7 1231       59        1 SensorReadFailed(SensorId(0xe), I2cError(NoRegister))
   8 1496       59      126 ControlPwm(0x0)
   9 1231       59        1 SensorReadFailed(SensorId(0x1d), CorruptReply)
  10 1496       59        1 ControlPwm(0x0)
  11 1231       59        1 SensorReadFailed(SensorId(0x1d), CorruptReply)
  12 1496       59        8 ControlPwm(0x0)
  13 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  14 1496       59        1 ControlPwm(0x0)
  15 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  16 1496       59       47 ControlPwm(0x0)
  17 1231       59        1 SensorReadFailed(SensorId(0xe), CorruptReply)
  18 1496       59        1 ControlPwm(0x0)
  19 1231       59        1 SensorReadFailed(SensorId(0xe), CorruptReply)
  20 1496       59       24 ControlPwm(0x0)
  21 1204       59        1 MiscReadFailed(SensorId(0x32), I2cError(NoRegister))
  22 1496       59        8 ControlPwm(0x0)
  23 1269       59        1 PowerModeChanged(PowerBitmask(0b1))
  24 1124       59        1 AutoState(Boot)
  25 1532       59        1 AutoState(Running)
  26 1496       59      359 ControlPwm(0x0)

and when we saw a similar event on mb-1 yesterday, we saw:

eliza@castle ~ $ humility -d /staff/luqman/fans-go/hubris.core.0 ringbuf thermal
humility: attached to dump
humility: ring buffer drv_i2c_devices::emc2305::__RINGBUF in thermal:
humility: ring buffer drv_i2c_devices::max31790::__RINGBUF in thermal:
humility: ring buffer task_thermal::__RINGBUF in thermal:
   TOTAL VARIANT
 1381043 ControlPwm
     192 SensorReadFailed
      56 AutoState(Boot)
      55 AutoState(Running)
      55 PowerModeChanged
      44 MiscReadFailed
       6 FanAdded
       1 Start
       1 ThermalMode(Auto)
       1 FanControllerInitialized
       1 SetFanWatchdogOk
 NDX LINE      GEN    COUNT PAYLOAD
  27 1231       58        1 SensorReadFailed(SensorId(0x9), CorruptReply)
  28 1496       58       25 ControlPwm(0x0)
  29 1231       58        1 SensorReadFailed(SensorId(0x13), CorruptReply)
  30 1231       58        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  31 1496       58        1 ControlPwm(0x0)
   0 1231       59        1 SensorReadFailed(SensorId(0x13), CorruptReply)
   1 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
   2 1496       59       11 ControlPwm(0x0)
   3 1231       59        1 SensorReadFailed(SensorId(0x27), CorruptReply)
   4 1496       59        1 ControlPwm(0x0)
   5 1231       59        1 SensorReadFailed(SensorId(0x27), CorruptReply)
   6 1496       59       29 ControlPwm(0x0)
   7 1231       59        1 SensorReadFailed(SensorId(0xe), I2cError(NoRegister))
   8 1496       59      126 ControlPwm(0x0)
   9 1231       59        1 SensorReadFailed(SensorId(0x1d), CorruptReply)
  10 1496       59        1 ControlPwm(0x0)
  11 1231       59        1 SensorReadFailed(SensorId(0x1d), CorruptReply)
  12 1496       59        8 ControlPwm(0x0)
  13 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  14 1496       59        1 ControlPwm(0x0)
  15 1231       59        1 SensorReadFailed(SensorId(0x18), CorruptReply)
  16 1496       59       47 ControlPwm(0x0)
  17 1231       59        1 SensorReadFailed(SensorId(0xe), CorruptReply)
  18 1496       59        1 ControlPwm(0x0)
  19 1231       59        1 SensorReadFailed(SensorId(0xe), CorruptReply)
  20 1496       59       24 ControlPwm(0x0)
  21 1204       59        1 MiscReadFailed(SensorId(0x32), I2cError(NoRegister))
  22 1496       59        8 ControlPwm(0x0)
  23 1269       59        1 PowerModeChanged(PowerBitmask(0b1))
  24 1124       59        1 AutoState(Boot)
  25 1532       59        1 AutoState(Running)
  26 1496       59      359 ControlPwm(0x0)

There's a dump from cosmo-os after today's event in /staff/luqman/fans-go/hubris.core.0, and one from mb-1 yesterday in in /staff/alan/mb-1-dump-eliza-this-is-the-dump. In both cases, the dumps were taken after the fan event and may be missing some state.

I wonder if we are either seeing some kind of general I2C jankiness that is also effecting our ability to feed the fan controller watchdog, or if we are extrapolating a worst case temperature that pushes the fan speed up because of a prolonged inability to get data from a particular sensor. Note that in both cases, we never actually made it into the "overheating" control state in thermal.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the thermal ring-buffer output from /staff/luqman/fans-go/hubris.core.0 and /staff/alan/mb-1-dump-eliza-this-is-the-dump, using the shown humility ... ringbuf thermal command. Trace the CorruptReply, NoRegister, ControlPwm, watchdog, and thermal-state events to determine whether I2C failures or sensor extrapolation caused the fan behavior. Done means identifying the cause and documenting or fixing it with evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.