oxidecomputer / oxidecomputer/omicron

Nexuses can duel when updating the `RackNetworkConfig` in the bootstore

Open
#10,855 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

The sync_switch_configuration background task is responsible for ensuring that the contents of the replicated bootstore contain an up-to-date SystemNetworkingConfig, which itself is comprised of a RackNetworkConfig (containing all the information required for upstream connectivity, including static routes / BGP / BFD configs) and a BlueprintExternalNetworkingConfig (containing all the information required to set up NAT entries for critical control plane services like boundary NTP). The high-level process here is "load the latest config from CRDB, then check whether it's newer than what's in the bootstore, and if so, update the bootstore". However, the RackNetworkConfig portion of this config has no way to know what "newer" means. If two Nexus instances are executing this background task at the same time as an operator changes the network config, this ordering is possible:

  1. Nexus A loads the current network config from CRDB (call this config C1)
  2. Operator makes a change, producing config C2
  3. Nexus B loads the current network config from CRDB (now C2)
  4. Nexus B checks the bootstore, finds C1, and updates the bootstore to hold C2
  5. Nexus A checks the bootstore, finds C2, and updates the bootstore to hold C1

This will correct itself the next time the bg task runs, so is eventually consistent, but we shouldn't allow the visible "flapping" produced by step 5: Nexus A should have a way of knowing that C2 is newer than the C1 it loaded and therefore it shouldn't update the bootstore.

When I initially added the BlueprintExternalNetworkingConfig it had the same problem, and as part of addressing it (#10320, which also describes this issue in less detail), we added a generation number from the blueprint to the bootstore config, specifically so Nexus can check that generation against the generation of the blueprint it loaded, and skip updating the bootstore if it finds itself in the blueprint version of step 5 above: https://github.com/oxidecomputer/omicron/blob/8f03979b59bf73f8de363bf54a6ccec111752c53/nexus/src/app/background/tasks/sync_switch_configuration.rs#L2123-L2132

The network config apparatus as a whole needs something akin to this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/src/app/background/tasks/sync_switch_configuration.rs, especially the bootstore update logic around the referenced lines, and read issue #10320 for the existing generation-check approach. Trace how RackNetworkConfig is loaded and compared, then verify that concurrent tasks cannot overwrite a newer bootstore configuration with an older one.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.