oxidecomputer / oxidecomputer/omicron
Crucible must be able to activate with 2/3 downstairs redundancy
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Currently, when a Crucible upstairs is activating downstairs, it blocks until all three downstairs have responded.
This requires perfect availability of all downstairs disks for instances to be started, which is especially problematic in the live update case. In the case where we pick any single sled to be updated, we migrate all instances off that sled, and we reboot it: for the duration of that sled being updated:
- If any instance across the fleet has a downstairs on any of the disks on the sled-under-update are trying to start...
- They won't see an ACK from the downstairs on the sled-under-update -- so they'll hang for the entire duration of the update
This is rough - it'll be a user-visible lack-of-availability, and will continue happening for different instances as we proceed with update across sleds in the rack.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating Crucible's upstairs activation and downstairs ACK handling, then trace the response-counting path; done means activation proceeds with two of three downstairs responses and does not wait for a downstairs on a sled undergoing live update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100