oxidecomputer / oxidecomputer/hubris
Reduce techports flapping as a result of Monorail 10G link watchdog
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
The Monorail task will periodically restart if the 10G link with Tofino is not up https://github.com/oxidecomputer/hubris/blob/b2b4861c32716d76bfa80243de87bbb4c5ce7f27/task/monorail-server/src/bsp/sidecar_bc.rs#L476-L502 in an attempt to work around the problem described here. But when the watchdog restarts the task it causes the front technician ports to flap, which is confusing during rack integration testing. This watchdog should really only be active if the PCIe link with Gimlet is attached and ideally would only kick in if Dendrite is running.
The sequencer can use the I2C link with Tofino to poll the state of its PCIe controller, which allows narrowing the window during which the watchdog is active. In addition, explore if monitoring some Tofino state registers can be used to determine if Dendrite is running.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in task/monorail-server/src/bsp/sidecar_bc.rs around lines 476-502 and read the linked document describing the 10G link problem. Trace how the sequencer can poll Tofino's PCIe controller over I2C and investigate whether Tofino state registers indicate that Dendrite is running. Done means the watchdog is narrowed to the attached PCIe link, ideally only while Dendrite runs, without causing front technician ports to flap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100