Possible `i2c_driver` hang on BRM42220026

Dumps are in /staff/core/rack2/BRM42220026/20240226/hubris.core.{0, 1}

@leftwo noticed the fans spinning up on this Gimlet. It looks like many tasks are hung waiting for the i2c_driver.

system time = 1207776588
ID TASK                       GEN PRI STATE
 0 jefe                         0   0 recv, notif: fault timer(T+12)
 1 net                          0   5 recv, notif: eth-irq(irq61) wake-timer(T+62)
 2 sys                          0   1 recv
 3 spi2_driver                  0   3 recv
 4 i2c_driver                   0   3 notif: i2c2-irq(irq33/irq34)
 5 spd                          0   2 notif: i2c1-irq(irq31/irq32)
 6 packrat                      0   1 recv
 7 thermal                      0   5 wait: send to i2c_driver/gen0
 8 power                        0   6 wait: send to gimlet_seq/gen0
 9 hiffy                        0   5 notif: bit31(T+171)
10 gimlet_seq                   0   4 wait: send to i2c_driver/gen0
11 hash_driver                  0   2 recv
12 hf                           0   3 recv
13 update_server                0   3 recv
14 sensor                       0   4 recv, notif: timer(T+419)
15 host_sp_comms                0   7 wait: reply from gimlet_seq/gen0
16 udpecho                      0   6 notif: socket
17 udpbroadcast                 0   6 notif: bit31(T+57)
18 control_plane_agent          0   6 wait: reply from validate/gen0
19 sprot                        0   4 notif: bit31(T+1)
20 validate                     0   5 wait: reply from i2c_driver/gen0
21 vpd                          0   4 recv
22 user_leds                    0   2 recv, notif: timer
23 dump_agent                   0   6 wait: reply from sprot/gen0
24 sbrmi                        0   4 recv
25 idle                         0   8 RUNNING

The i2c_driver is just chilling out, waiting for a notification:

matt@jeeves ~ () $ h -d /staff/core/rack2/BRM42220026/20240226/hubris.core.0 tasks -sl i2c_driver
humility: attached to dump
system time = 1207776588
ID TASK                       GEN PRI STATE
 4 i2c_driver                   0   3 notif: i2c2-irq(irq33/irq34)
   |
   +--->  0x2403b200 0x080aa9c0 userlib::sys_recv_stub
                     @ /hubris/sys/userlib/src/lib.rs:332
          0x2403b220 0x080a809c core::ops::function::FnOnce::call_once
                     @ /rustc/95a3a7277b44bbd2dd3485703d9a05f64652b60e/library/core/src/ops/function.rs:251
          0x2403b380 0x080a8e1e drv_stm32xx_i2c::I2cController::write_read
                     @ /hubris/drv/stm32xx-i2c/src/lib.rs:534
          0x2403b380 0x080a8a2a drv_stm32xx_i2c_server::main::{{closure}}
                     @ /hubris/drv/stm32xx-i2c-server/src/main.rs:358
          0x2403b380 0x080a8a2a userlib::hl::recv_without_notification::{{closure}}
                     @ /hubris/sys/userlib/src/hl.rs:130
          0x2403b380 0x080a89cc userlib::hl::recv
                     @ /hubris/sys/userlib/src/hl.rs:83
          0x2403b380 0x080a89cc userlib::hl::recv_without_notification
                     @ /hubris/sys/userlib/src/hl.rs:123
          0x2403b380 0x080a8ef4 main
                     @ /hubris/drv/stm32xx-i2c-server/src/main.rs:322

Taking two dumps, there's no change to i2c_driver's ringbufs, so it's seemingly not making forward progress.

The ringbufs indicate that it's Very Mad:

 NDX LINE      GEN    COUNT PAYLOAD
  ... a few dozen lines elided...
 144  262      836        1 ResetMux(0x73)
 145  392      836        1 MuxError(BusLockedMux)
 146  252      836        1 Reset((I2C2, PortIndex(0x0)))
 147  262      836        1 ResetMux(0x73)
 148  392      836        1 MuxError(BusLockedMux)
 149  252      836        1 Reset((I2C2, PortIndex(0x0)))
 150  262      836        1 ResetMux(0x73)
 151  190      836        1 MuxUnknownRecover((I2C2, PortIndex(0x0)))
 152  392      836        1 MuxError(BusLockedMux)
 153  252      836        1 Reset((I2C2, PortIndex(0x0)))
 154  262      836        1 ResetMux(0x73)
 155  392      836        1 MuxError(BusLockedMux)
 156  252      836        1 Reset((I2C2, PortIndex(0x0)))
 157  262      836        1 ResetMux(0x73)
 158  392      836        1 MuxError(BusLockedMux)
 159  252      836        1 Reset((I2C2, PortIndex(0x0)))
 160  262      836        1 ResetMux(0x73)
 161  190      836        1 MuxUnknownRecover((I2C2, PortIndex(0x0)))

As far as I can tell, the only mux at address 0x73 is the M.2 mux:

[[config.i2c.devices]]
bus = "m2"
address = 0x73
device = "pca9545"
description = "M.2 mux"
refdes = "U422"

I2C2, PortIndex(0x0) matches the M.2 bus in the manifest:

#
# I2C2: Front/M.2 bus
#
[[config.i2c.controllers]]
controller = 2

#
# SMBUS_SP_TO_M2_SMCLK_A2_V3P3
# SMBUS_SP_TO_M2_SMDAT_A2_V3P3
#
[config.i2c.controllers.ports.B]
name = "m2"
description = "M.2 bus"
scl.pin = 10
sda.pin = 11
af = 4