oxidecomputer / oxidecomputer/hubris
Sidecar sequencer panics as recovery when front I/O not programmed.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
I think we'd like to get some better error messages here than just a panic, especially since debugging this in MFG is basically impossible today.
We get in the station logs:
humility: attached to 0483:3754:0039001D4741500920383733 via ST-Link V3
Sequencer.set_tofino_seq_policy() => Err(<server died; its new ID is 26>)
manually debugging, we set the state disposition to hold:
humility jefe -H sequencer
gets us
humility tasks
D TASK GEN PRI STATE
0 jefe 0 0 recv, notif: fault timer(T+12)
1 sys 0 1 recv, notif: exti-wildcard-irq(irq6/irq7/irq8/irq9/irq10/irq23/irq40)
2 rng_driver 0 6 recv
3 update_server 0 3 recv
4 auxflash 0 3 recv
5 net 0 5 wait: send to sequencer/gen16
6 control_plane_agent 0 7 wait: send to net/gen0
7 sprot 0 4 recv
8 udpecho 0 6 wait: send to net/gen0
9 udpbroadcast 0 6 wait: send to net/gen0
10 monorail 0 6 wait: send to sequencer/gen16
11 i2c_driver 0 2 recv
12 hiffy 0 5 notif: bit31(T+209)
13 sensor 0 4 recv
14 ecp5_mainboard 0 3 recv
15 ecp5_front_io 0 3 recv
16 transceivers 0 6 wait: send to net/gen0
17 packrat 0 3 recv
18 sequencer 1168 4 FAULT: panicked at drv/sidecar-seq-server/src/main.rs:963:35:
explicit panic (was: ready)
19 thermal 0 5 wait: send to sequencer/gen16
20 power 390 6 wait: send to sequencer/gen16
21 validate 0 5 recv
22 ignition 0 5 wait: send to sequencer/gen16
23 vpd 0 3 recv
24 dump_agent 0 6 wait: send to net/gen0
25 idle 0 8 RUNNING
26 udprpc 0 6 wait: send to net/gen0
so the panic comes from here: front_io_board.init().unwrap_lite();
and the sequencer ringbuff shows:
humility: attached to 0483:3754:0039001D4741500920383733 via ST-Link V3
humility: ring buffer drv_oxide_vpd::__RINGBUF in sequencer:
humility: ring buffer drv_packrat_vpd_loader::__RINGBUF in sequencer:
humility: ring buffer drv_sidecar_seq_server::__RINGBUF in sequencer:
NDX LINE GEN COUNT PAYLOAD
0 825 1 1 FpgaInit
1 868 1 1 SkipLoadingBitstream
2 878 1 1 MainboardControllerId(0x1de5bae)
3 892 1 1 MainboardControllerChecksum(0x6407475e)
4 924 1 1 MainboardControllerVersion(0x283)
5 925 1 1 MainboardControllerSha(0x3c8d1c33)
6 926 1 1 FpgaInitComplete
7 31 1 1 LoadingClockConfiguration
8 951 1 1 ClockConfigurationComplete
9 956 1 1 FrontIOBoardPresent
10 81 1 1 LoadingFrontIOControllerBitstream { fpga_id: 0x0 }
11 86 1 1 FpgaBitstreamError(0x300)
This ended up being a broken ARF6 cable between main and the front-io but this is an extremely annoying issue to debug at the MFG station so getting some better visibility/ error reporting here will help.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in drv/sidecar-seq-server/src/main.rs at the front_io_board.init().unwrap_lite() call around line 963. Read the sequencer ring-buffer events showing FrontIOBoardPresent and FpgaBitstreamError, then trace how this initialization failure is reported. Done means a failed front-I/O initialization provides useful diagnostic error information instead of only panicking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100