dump-agent gets stuck in send to net

On a system with a whole lot of dumps (e.g. the gimlet from #1779, which has a thermal task that's crashed 7483 times), running humility dump -l can time out. When this occurs, I've seen the hiffy task left in a bad state where no other commands that use hiffy can succeed.

For instance:

# running a humility i2c command, which uses hiffy, works fine
eliza@niles ~ $ pfexec humility -t gimlet-c i2c -b front --scan
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3

Device scan on controller I2C2:

    R = Reserved   - = No device   \o/ = Device found   X = Timed out

ADDR     0x0 0x1 0x2 0x3 0x4 0x5 0x6 0x7 0x8 0x9 0xa 0xb 0xc 0xd 0xe 0xf
0x00       R   R   R   R   R   R   R   R   -   -   -   -   -   -   -   -
0x10       -   -   -   -   -   -   -   -   -   -   -   -   -   -   -   -
0x20       -   -   -   -   -   -   -   -   -   -   -   -   -   -   -   -
0x30       -   -   -   -   -   -   -   -   -   -   -   -   -   -   -   -
0x40       -   -   -   -   -   -   -   - \o/   -   -   -   -   -   -   -
0x50     \o/ \o/ \o/ \o/   -   -   -   - \o/ \o/ \o/ \o/   -   -   -   -
0x60       -   -   -   -   -   -   -   -   -   -   -   -   -   -   -   -
0x70       -   -   -   -   -   -   -   -   -   -   -   -   R   R   R   R

# now, i try listing dumps, which times out
eliza@niles ~ $ pfexec humility -t gimlet-c dump -l
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
humility: using hiffy dump agent
AREA TASK                  TIME       SIZE
humility dump failed: operation timed out

# running the same humility i2c command now fails
eliza@niles ~ $ pfexec humility -t gimlet-c i2c -b front --scan
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
humility i2c failed: HIF execution facility unavailable

# note that hiffy is stuck in recv from dump-agent, which is in send to net
eliza@niles ~ $ pfexec humility -t gimlet-c tasks
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
system time = 4215901
ID TASK                       GEN PRI STATE
 0 jefe                         0   0 recv, notif: fault timer(T+101)
 1 net                          0   5 notif: jefe-state-change
 2 sys                          0   1 recv, notif: exti-wildcard-irq(irq6/irq7/irq8/irq9/irq10/irq23/irq40)
 3 spi2_driver                  0   3 recv(gimlet_seq/gen0 only)
 4 i2c_driver                   0   3 recv
 5 spd                          0   2 notif: jefe-state-change
 6 packrat                      0   1 recv
 7 thermal                   7483   5 FAULT: panicked at task/thermal/src/bsp/gimlet_bcdef.rs:162:28:
explicit panic (was: ready)
 8 power                        0   6 wait: send to gimlet_seq/gen0
 9 hiffy                        1   5 wait: send to dump_agent/gen0
10 gimlet_seq                   0   4 notif: bit31(T+3)
11 gimlet_inspector             0   6 wait: send to net/gen0
12 hash_driver                  0   2 recv
13 hf                           0   3 notif: bit31(T+330)
14 update_server                0   3 recv
15 sensor                       0   4 recv
16 host_sp_comms                0   7 recv, notif: jefe-state-change usart-irq(irq82) multitimer control-plane-agent
17 udpecho                      0   6 wait: send to net/gen0
18 udpbroadcast                 0   6 wait: send to net/gen0
19 control_plane_agent          0   6 wait: send to net/gen0
20 sprot                        0   4 recv
21 validate                     0   5 recv
22 vpd                          0   4 recv
23 user_leds                    0   2 recv, notif: timer
24 dump_agent                   0   6 wait: send to net/gen0
25 sbrmi                        0   4 recv
26 idle                         0   8 RUNNING
27 udprpc                       0   6 wait: send to net/gen0

manually restarting the hiffy task using humility jefe -fault hiffy and humility jefe --release hiffy, seems to unstick it, until the next time that you run humility dump -l.

Reproduce the failure with humility dump -l on a system with many dumps, then inspect the hiffy and dump_agent task states using humility tasks; the report shows both blocked while sending to net. Trace the dump-agent request path and its interaction with hiffy, and verify that a timed-out dump no longer leaves hiffy unavailable by rerunning the I2C scan afterward.