oxidecomputer / oxidecomputer/omicron
BFD configuration provided at rack setup time is lost in the handoff to Nexus
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
When Nexus gets the handoff from RSS, the handoff request includes the full RackNetworkConfig provided by the operator to RSS. Nexus does not destructure that, and instead picks apart individual fields; e.g., setting up the initial address lot: https://github.com/oxidecomputer/omicron/blob/a3012e0080d7aca311e2dfe2f45cba7ae828ca5b/nexus/src/app/rack.rs#L318-L321
and any BGP config: https://github.com/oxidecomputer/omicron/blob/a3012e0080d7aca311e2dfe2f45cba7ae828ca5b/nexus/src/app/rack.rs#L344
However, nowhere in here does it access rack_networking_config.bfd, which I believe means we'd see something like this:
- The
RackNetworkConfigas provided by the operator at RSS time would be written to the bootstore during RSS - Any BFD information included in the RSS config would be used during the RSS network setup, because sled-agent does read this information and apply it in its "early networking" path
- During handoff, Nexus would not persist this BFD config into cockroach
- Therefore, the first time the
sync_switch_configurationtask ran after RSS, it would construct a newRackNetworkConfigwith no BFD configuration at all, and would overwrite the bootstoreRackNetworkConfigset up by RSS, as well as actively tearing down any BFD sessions set up during RSS
I ran into this while attempting to test some maghemite BFD changes in a4x2, where following these instructions produced a setup where the ipv6 links had configured BFD peers and the ipv4 links did not, even though the ipv4 links' BFD configs were present in the RSS config toml.
I tried to see if this was lost along the way, but I think maybe it has never worked? #4852 was the initial BFD support PR, but it didn't include RSS or sled-agent support, only configuration via the external API after the rack was set up. #5313 went back and added it to the RSS config, bootstore, and sled-agent, but didn't touch the RSS -> Nexus handoff path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in nexus/src/app/rack.rs at the RSS-to-Nexus handoff, especially where RackNetworkConfig fields and BGP configuration are extracted. Trace how rack_networking_config.bfd should reach bootstore and the sync_switch_configuration task. Done means BFD supplied in RSS configuration is persisted through handoff and is not removed when synchronization runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100