oxidecomputer / oxidecomputer/hubris

`net` not initializing itself makes everyone else wait on it

Open
#2,183 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Triaged H2'26
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

When debugging mfg-troubleshooting#110, we grew suspicious of the 2.5V PHY rail not being up. The failure mode here was that we timed-out when try to do a hiffy -c ControlPLaneAgent.set_startup_options. The fact that we seemed otherwise fine was strange, but looking at tasks, everyone was waiting on net

$ pfexec humility tasks
humility: attached to 0483:3754:004E001F3133510337363734 via ST-Link V3
system time = 4415
ID TASK                       GEN PRI STATE
 0 jefe                         0   0 recv, notif: fault timer(T+85)
 1 net                          0   5 notif: bit31(T+3)
 2 sys                          0   1 recv, notif: exti-wildcard-irq(irq6/irq7/irq8/irq9/irq10/irq23/irq40)
 3 spi2_driver                  0   3 recv
 4 i2c_driver                   0   3 recv
 5 spd                          0   2 notif: i2c1-irq(irq31/irq32)
 6 packrat                      0   1 recv
 7 thermal                      0   5 recv, notif: timer(T+858)
 8 power                        0   6 recv, notif: timer(T+864)
 9 hiffy                        0   5 notif: bit31(T+250)
10 gimlet_seq                   0   4 recv, notif: timer vcore
11 gimlet_inspector             0   6 wait: send to net/gen0
12 hash_driver                  0   2 recv
13 hf                           0   3 recv
14 update_server                0   3 recv
15 sensor                       0   4 recv
16 host_sp_comms                0   8 recv, notif: jefe-state-change usart-irq(irq82) multitimer(T+37) control-plane-agent
17 udpecho                      0   6 wait: send to net/gen0
18 udpbroadcast                 0   6 wait: send to net/gen0
19 control_plane_agent          0   7 wait: send to net/gen0
20 sprot                        0   4 recv
21 validate                     0   5 recv
22 vpd                          0   4 recv
23 user_leds                    0   2 recv, notif: timer
24 dump_agent                   0   6 wait: send to net/gen0
25 sbrmi                        0   4 recv
26 idle                         0   9 RUNNING
27 udprpc                       0   6 wait: send to net/gen0

net is spending a lot of time in crappy_spin_until it seems:

$ pfexec humility tasks -sl net
humility: attached to 0483:3754:004E001F3133510337363734 via ST-Link V3
system time = 83751
ID TASK                       GEN PRI STATE
 1 net                          0   5 notif: bit31(T+3)
   |
   +--->  0x24010f60 0x0807bc9a userlib::sys_recv_stub
                     @ /hubris/sys/userlib/src/lib.rs:368
          0x24010fa8 0x0807bd20 userlib::sys_get_timer
                     @ /hubris/sys/userlib/src/lib.rs:1130
          0x24010fa8 0x0807bcf8 userlib::hl::sleep_until
                     @ /hubris/sys/userlib/src/hl.rs:425
          0x24010fa8 0x0807bd20 userlib::hl::sleep_for
                     @ /hubris/sys/userlib/src/hl.rs:459
          0x24011f40 0x08072934 drv_stm32h7_eth::crappy_spin_until
                     @ /hubris/drv/stm32h7-eth/src/lib.rs:59
          0x24011f40 0x08072942 main
                     @ /hubris/task/net/src/main.rs:163

If the PHY isn't powered/responding, we seem to just be stuck in this indefinitely.

We should make this sort of problem fail louder or perhaps more gracefully.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with drv/stm32h7-eth/src/lib.rs:59 and task/net/src/main.rs:163, where net enters crappy_spin_until while waiting for the PHY. Reproduce or trace startup with an unpowered or unresponsive PHY and determine how the failure is reported. Done means net no longer waits indefinitely and the failure is surfaced or handled gracefully.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.