oxidecomputer / oxidecomputer/hubris

monorail: Reconsider bf_sde workaround(s)

Open
#2,671 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

We currently have a workaround in monorail that determines if the network link between the tofino (main network switch) and vsc7448 (management network switch) does not properly autonegotiate:

https://github.com/oxidecomputer/hubris/blob/1cbf835c1db3c8d9a7a53274e5af38e002c1c244/task/monorail-server/src/bsp/sidecar_bcd.rs#L559-L598

In a few observed cases, CC cs#1485, this failure to autonegotiate is potentially due to the PCIe link between the scrimlet and tofino not operating nominally (and therefore not being a good autonegotiation partner for the vsc7448), rather than some unexpected behavior from the vsc7448 itself.

However, with this workaround, we completely re-sequence the vsc7448 every 20 seconds, meaning that the management network will be brought down and up again repeatedly until the link between the vsc7448 and tofino is resolved.

Although the link between the scrimlet and its management network won't be functional in this case (as it connects through the tofino, so no tofino <-> vsc7448 link means no access to the management network), we are also inhibiting tech-port access to the management network on that switch.

Unfortunately, as the scrimlet does not have any means of directly talking to its sidecar's SP, it's currently hard for us to know if the failure to negotiate is ephemeral (and we should keep retrying), or if it is more permanent (requiring a tofino re-sequence). We may want to make this negotiation process bounded, or with an increasing back-off.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with task/monorail-server/src/bsp/sidecar_bcd.rs around lines 559-598 and trace how the bf_sde workaround detects autonegotiation failure and re-sequences the vsc7448. Determine whether retrying should be bounded or use increasing back-off, then verify that the management network is not repeatedly disrupted while the tofino-to-vsc7448 link remains unresolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, networking
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.