oxidecomputer / oxidecomputer/omicron

self-service update speedup: combine host reboot for OS and SP updates

Open
#9,460 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

A big part of overall self-service update time is the sled reboots that we do. Currently, we do them twice:

  • for SP updates: these tend to take about 5 minutes in the end-to-end update time
  • for host OS updates: these tend to take about 8 minutes in the end-to-end update time

(The reason for the difference is that the planner does not wait for the host OS to become available before proceeding with the next SP update, but it does do that before proceeding with the next host OS update.)

In an ideal world, when we know we're doing both an SP and host OS update, we would skip the host reset for the SP update, knowing we're about to do a host OS update that will reset the SP. This is tricky for a few reasons:

  • Correctness here relies on something the planner will be doing in the future, and introduces the possibility that we stage an SP update but don't apply it for whatever reason.
  • It's possible that things change or get interrupted in between these steps. Might we then need to configure an SP reset by itself?
  • We need to continue supporting both paths because sometimes there is no SP update (and conceivably there could be no host OS update, though that seems unlikely).

The upside is a little tricky to summarize.

  • The "SP update" step could be quite a bit faster but it's not clear how much. Maybe 5 minutes goes to 1 minute? That's a savings of about 64 minutes on a half-rack. But that's a guess.
  • Instances would be bounced half as much as they are today (which is kind of a big deal, given the current impact on the instance each time).
  • This might also eliminate the problem we have where we might do an SP update, then move onto the next (or do a Crucible zone update) before the host is back, resulting in two downstairs being offline for some disks.

In terms of intrinsic risks: hopefully all of the above tricky cases are testable with planner tests. There may be other risks depending on how we wind up implementing this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the planner's update sequencing and existing planner tests, focusing on how SP and host OS updates currently schedule sled reboots. Trace the interruption and no-update paths described in the issue. Done means the combined-update behavior and separate-update paths are covered by tests without leaving a staged SP update unapplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.