oxidecomputer / oxidecomputer/omicron
Nexuses can duel when updating the `RackNetworkConfig` in the bootstore
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
The sync_switch_configuration background task is responsible for ensuring that the contents of the replicated bootstore contain an up-to-date SystemNetworkingConfig, which itself is comprised of a RackNetworkConfig (containing all the information required for upstream connectivity, including static routes / BGP / BFD configs) and a BlueprintExternalNetworkingConfig (containing all the information required to set up NAT entries for critical control plane services like boundary NTP). The high-level process here is "load the latest config from CRDB, then check whether it's newer than what's in the bootstore, and if so, update the bootstore". However, the RackNetworkConfig portion of this config has no way to know what "newer" means. If two Nexus instances are executing this background task at the same time as an operator changes the network config, this ordering is possible:
- Nexus A loads the current network config from CRDB (call this config C1)
- Operator makes a change, producing config C2
- Nexus B loads the current network config from CRDB (now C2)
- Nexus B checks the bootstore, finds C1, and updates the bootstore to hold C2
- Nexus A checks the bootstore, finds C2, and updates the bootstore to hold C1
This will correct itself the next time the bg task runs, so is eventually consistent, but we shouldn't allow the visible "flapping" produced by step 5: Nexus A should have a way of knowing that C2 is newer than the C1 it loaded and therefore it shouldn't update the bootstore.
When I initially added the BlueprintExternalNetworkingConfig it had the same problem, and as part of addressing it (#10320, which also describes this issue in less detail), we added a generation number from the blueprint to the bootstore config, specifically so Nexus can check that generation against the generation of the blueprint it loaded, and skip updating the bootstore if it finds itself in the blueprint version of step 5 above: https://github.com/oxidecomputer/omicron/blob/8f03979b59bf73f8de363bf54a6ccec111752c53/nexus/src/app/background/tasks/sync_switch_configuration.rs#L2123-L2132
The network config apparatus as a whole needs something akin to this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in nexus/src/app/background/tasks/sync_switch_configuration.rs, especially the bootstore update logic around the referenced lines, and read issue #10320 for the existing generation-check approach. Trace how RackNetworkConfig is loaded and compared, then verify that concurrent tasks cannot overwrite a newer bootstore configuration with an older one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100