oxidecomputer / oxidecomputer/omicron
Persist RSS completed marker on all sleds
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Right now, when we complete RSS we persist a marker file (ledger) that indicates that RSS should not be re-run as it has already completed. This prevents data loss. However, this marker file is only present on the M.2 devices attached to the scrimlet where RSS was run. If RSS runs on the other scrimlet, or one of the sleds is swapped into scimlet position, the marker will not be present.
We want this marker to be present on all sleds. We could require it to be persisted to all sleds before completing RSS by direct copying. However, this means that if a new sled is added to the rack and put in the scrimlet position it will not have the files. We already have a mechanism to gossip around configuration required for early boot: namely the bootstore. We can put this marker key in the bootstore as well and it will propagate asynchronously over the bootstrap network to any sled in the rack. Max delay for an online sled is ~1 second.
This will require some changes to the bootstore to support multiple keys, but this is not unreasonable, and something we have considered in the past.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing where RSS completion writes its marker ledger and how the bootstore currently handles configuration keys. Define the bootstore changes needed for multiple keys, then verify that the completion marker propagates to every online sled and prevents RSS from rerunning after sled swaps or new sleds joining the rack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100