oxidecomputer / oxidecomputer/hubris
monorail: Reconsider bf_sde workaround(s)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
We currently have a workaround in monorail that determines if the network link between the tofino (main network switch) and vsc7448 (management network switch) does not properly autonegotiate:
In a few observed cases, CC cs#1485, this failure to autonegotiate is potentially due to the PCIe link between the scrimlet and tofino not operating nominally (and therefore not being a good autonegotiation partner for the vsc7448), rather than some unexpected behavior from the vsc7448 itself.
However, with this workaround, we completely re-sequence the vsc7448 every 20 seconds, meaning that the management network will be brought down and up again repeatedly until the link between the vsc7448 and tofino is resolved.
Although the link between the scrimlet and its management network won't be functional in this case (as it connects through the tofino, so no tofino <-> vsc7448 link means no access to the management network), we are also inhibiting tech-port access to the management network on that switch.
Unfortunately, as the scrimlet does not have any means of directly talking to its sidecar's SP, it's currently hard for us to know if the failure to negotiate is ephemeral (and we should keep retrying), or if it is more permanent (requiring a tofino re-sequence). We may want to make this negotiation process bounded, or with an increasing back-off.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with task/monorail-server/src/bsp/sidecar_bcd.rs around lines 559-598 and trace how the bf_sde workaround detects autonegotiation failure and re-sequences the vsc7448. Determine whether retrying should be bounded or use increasing back-off, then verify that the management network is not repeatedly disrupted while the tofino-to-vsc7448 link remains unresolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot, networking
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100