oxidecomputer / oxidecomputer/omicron

Persist RSS completed marker on all sleds

Open
#7,186 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bootstrap services Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Right now, when we complete RSS we persist a marker file (ledger) that indicates that RSS should not be re-run as it has already completed. This prevents data loss. However, this marker file is only present on the M.2 devices attached to the scrimlet where RSS was run. If RSS runs on the other scrimlet, or one of the sleds is swapped into scimlet position, the marker will not be present.

We want this marker to be present on all sleds. We could require it to be persisted to all sleds before completing RSS by direct copying. However, this means that if a new sled is added to the rack and put in the scrimlet position it will not have the files. We already have a mechanism to gossip around configuration required for early boot: namely the bootstore. We can put this marker key in the bootstore as well and it will propagate asynchronously over the bootstrap network to any sled in the rack. Max delay for an online sled is ~1 second.

This will require some changes to the bootstore to support multiple keys, but this is not unreasonable, and something we have considered in the past.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing where RSS completion writes its marker ledger and how the bootstore currently handles configuration keys. Define the bootstore changes needed for multiple keys, then verify that the completion marker propagates to every online sled and prevents RSS from rerunning after sled swaps or new sleds joining the rack.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.