oxidecomputer / oxidecomputer/hubris

Sidecar sequencer panics as recovery when front I/O not programmed.

Open
#2,069 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

I think we'd like to get some better error messages here than just a panic, especially since debugging this in MFG is basically impossible today.

We get in the station logs:

humility: attached to 0483:3754:0039001D4741500920383733 via ST-Link V3
Sequencer.set_tofino_seq_policy() => Err(<server died; its new ID is 26>)

manually debugging, we set the state disposition to hold:
humility jefe -H sequencer

gets us

humility tasks
D TASK                       GEN PRI STATE
 0 jefe                         0   0 recv, notif: fault timer(T+12)
 1 sys                          0   1 recv, notif: exti-wildcard-irq(irq6/irq7/irq8/irq9/irq10/irq23/irq40)
 2 rng_driver                   0   6 recv
 3 update_server                0   3 recv
 4 auxflash                     0   3 recv
 5 net                          0   5 wait: send to sequencer/gen16
 6 control_plane_agent          0   7 wait: send to net/gen0
 7 sprot                        0   4 recv
 8 udpecho                      0   6 wait: send to net/gen0
 9 udpbroadcast                 0   6 wait: send to net/gen0
10 monorail                     0   6 wait: send to sequencer/gen16
11 i2c_driver                   0   2 recv
12 hiffy                        0   5 notif: bit31(T+209)
13 sensor                       0   4 recv
14 ecp5_mainboard               0   3 recv
15 ecp5_front_io                0   3 recv
16 transceivers                 0   6 wait: send to net/gen0
17 packrat                      0   3 recv
18 sequencer                 1168   4 FAULT: panicked at drv/sidecar-seq-server/src/main.rs:963:35:
explicit panic (was: ready)
19 thermal                      0   5 wait: send to sequencer/gen16
20 power                      390   6 wait: send to sequencer/gen16
21 validate                     0   5 recv
22 ignition                     0   5 wait: send to sequencer/gen16
23 vpd                          0   3 recv
24 dump_agent                   0   6 wait: send to net/gen0
25 idle                         0   8 RUNNING
26 udprpc                       0   6 wait: send to net/gen0

so the panic comes from here: front_io_board.init().unwrap_lite();

and the sequencer ringbuff shows:

humility: attached to 0483:3754:0039001D4741500920383733 via ST-Link V3
humility: ring buffer drv_oxide_vpd::__RINGBUF in sequencer:
humility: ring buffer drv_packrat_vpd_loader::__RINGBUF in sequencer:
humility: ring buffer drv_sidecar_seq_server::__RINGBUF in sequencer:
 NDX LINE      GEN    COUNT PAYLOAD
   0  825        1        1 FpgaInit
   1  868        1        1 SkipLoadingBitstream
   2  878        1        1 MainboardControllerId(0x1de5bae)
   3  892        1        1 MainboardControllerChecksum(0x6407475e)
   4  924        1        1 MainboardControllerVersion(0x283)
   5  925        1        1 MainboardControllerSha(0x3c8d1c33)
   6  926        1        1 FpgaInitComplete
   7   31        1        1 LoadingClockConfiguration
   8  951        1        1 ClockConfigurationComplete
   9  956        1        1 FrontIOBoardPresent
  10   81        1        1 LoadingFrontIOControllerBitstream { fpga_id: 0x0 }
  11   86        1        1 FpgaBitstreamError(0x300)

This ended up being a broken ARF6 cable between main and the front-io but this is an extremely annoying issue to debug at the MFG station so getting some better visibility/ error reporting here will help.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in drv/sidecar-seq-server/src/main.rs at the front_io_board.init().unwrap_lite() call around line 963. Read the sequencer ring-buffer events showing FrontIOBoardPresent and FpgaBitstreamError, then trace how this initialization failure is reported. Done means a failed front-I/O initialization provides useful diagnostic error information instead of only panicking.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.