oxidecomputer / oxidecomputer/hubris

Reduce techports flapping as a result of Monorail 10G link watchdog

Open
#1,509 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

The Monorail task will periodically restart if the 10G link with Tofino is not up https://github.com/oxidecomputer/hubris/blob/b2b4861c32716d76bfa80243de87bbb4c5ce7f27/task/monorail-server/src/bsp/sidecar_bc.rs#L476-L502 in an attempt to work around the problem described here. But when the watchdog restarts the task it causes the front technician ports to flap, which is confusing during rack integration testing. This watchdog should really only be active if the PCIe link with Gimlet is attached and ideally would only kick in if Dendrite is running.

The sequencer can use the I2C link with Tofino to poll the state of its PCIe controller, which allows narrowing the window during which the watchdog is active. In addition, explore if monitoring some Tofino state registers can be used to determine if Dendrite is running.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in task/monorail-server/src/bsp/sidecar_bc.rs around lines 476-502 and read the linked document describing the 10G link problem. Trace how the sequencer can poll Tofino's PCIe controller over I2C and investigate whether Tofino state registers indicate that Dendrite is running. Done means the watchdog is narrowed to the attached PCIe link, ideally only while Dendrite runs, without causing front technician ports to flap.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.