oxidecomputer / oxidecomputer/omicron
Quiesce sagas when parking the rack for updates
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Saga nodes are neither required nor generally expected to be compatible across different software release versions. Our current mupdate-driven process is to park the rack then mupdate all sleds. But if we park the rack while there are still live sagas, we could accidentally start up a new software release that tries to resume a saga that was serialized on the prior release.
One way to address this would be to add a manual step in the rack update process that checks for any sagas in such a state prior to performing the update. But this is kinda awkward: a running Nexus might start a new saga at any given time (e.g., kicked off by some RPW on a periodic timer), so a check for sagas would only be valid if all Nexus instances were stopped first. An optimistic attempt at this might look like:
- Shut down all Nexus instances
- Check for live sagas
- If any live sagas exist, restart the Nexus it's assigned to
- Once it completes, shut down that Nexus
- Check again to ensure no additional sagas were created while the prior one was running (repeat as needed)
It would probably be kinder to our updaters to build something into Nexus + omdb that allowed an updater to put Nexus into a "no new sagas are allowed to start; let me know when all existing sagas are complete" quiesced state. Then the update procedure could be updated to:
- Use omdb to quiesce Nexus sagas
- Park the rack
- Continue as we normally do
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Nexus and omdb saga lifecycle and the rack update procedure described in the issue. Define how Nexus enters a no-new-sagas state, reports completion of existing sagas, and how the update flow uses that state; done means the rack can be parked without resuming incompatible serialized sagas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100