oxidecomputer / oxidecomputer/omicron

BFD configuration provided at rack setup time is lost in the handoff to Nexus

Open
#10,886 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

networking
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When Nexus gets the handoff from RSS, the handoff request includes the full RackNetworkConfig provided by the operator to RSS. Nexus does not destructure that, and instead picks apart individual fields; e.g., setting up the initial address lot: https://github.com/oxidecomputer/omicron/blob/a3012e0080d7aca311e2dfe2f45cba7ae828ca5b/nexus/src/app/rack.rs#L318-L321

and any BGP config: https://github.com/oxidecomputer/omicron/blob/a3012e0080d7aca311e2dfe2f45cba7ae828ca5b/nexus/src/app/rack.rs#L344

However, nowhere in here does it access rack_networking_config.bfd, which I believe means we'd see something like this:

  • The RackNetworkConfig as provided by the operator at RSS time would be written to the bootstore during RSS
  • Any BFD information included in the RSS config would be used during the RSS network setup, because sled-agent does read this information and apply it in its "early networking" path
  • During handoff, Nexus would not persist this BFD config into cockroach
  • Therefore, the first time the sync_switch_configuration task ran after RSS, it would construct a new RackNetworkConfig with no BFD configuration at all, and would overwrite the bootstore RackNetworkConfig set up by RSS, as well as actively tearing down any BFD sessions set up during RSS

I ran into this while attempting to test some maghemite BFD changes in a4x2, where following these instructions produced a setup where the ipv6 links had configured BFD peers and the ipv4 links did not, even though the ipv4 links' BFD configs were present in the RSS config toml.

I tried to see if this was lost along the way, but I think maybe it has never worked? #4852 was the initial BFD support PR, but it didn't include RSS or sled-agent support, only configuration via the external API after the rack was set up. #5313 went back and added it to the RSS config, bootstore, and sled-agent, but didn't touch the RSS -> Nexus handoff path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/src/app/rack.rs at the RSS-to-Nexus handoff, especially where RackNetworkConfig fields and BGP configuration are extracted. Trace how rack_networking_config.bfd should reach bootstore and the sync_switch_configuration task. Done means BFD supplied in RSS configuration is persisted through handoff and is not removed when synchronization runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.