oxidecomputer / oxidecomputer/omicron

"sled add" could be more asynchronous

Open
#5,132 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Add/expunge sled
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Right now, the "add sled" external API call invokes a PUT /sleds API on some sled agent and waits for it to complete. That process in turn communicates with bootstrap agent on the new sled to start Sled Agent on the new sled. Aside from #5121, we expect this to be fairly quick. But things like #5121 do happen. We did #5116 as a band-aid, but it's not a great solution because it means an API call could block for several minutes. That's not a great pattern for HTTP for lots of reasons.

What if instead we moved this to Reconfigurator (blueprint planning/execution)? Proposal:

  • the "add sled" API immediately inserts a sled record with state uninitialized
  • during planning, if we see any sleds in state "uninitialized", we put them into a set of sleds_to_initialize: BTreeSet<Uuid> (or equivalent)
  • during blueprint execution, for each sled in that state, we invoke the PUT /sleds API on some sled agent

This assumes that the PUT /sleds request is idempotent and can be invoked concurrently against multiple sled agents for the same new sled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the “add sled” API and the Reconfigurator planning and blueprint execution paths named in the proposal. Trace how the sled record and its state are handled, then verify that the API returns without waiting and that uninitialized sleds are collected and initialized during planning and execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.