oxidecomputer / oxidecomputer/omicron

Sled agent tears down all running zones (including running VMs) if it crashes & restarts

Open
#2,646 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Repro steps: none yet; observed by inspection.

I've been looking around and can't find any mechanism by which a sled agent that dies and restarts is able to rediscover the Propolis zones running on the sled. The VMs might actually still survive and be reachable by users, but attempts to do anything with them in Nexus will either do nothing or move the instance into a Failed state (depending on the precise error code sled agent returns when Nexus asks to change the instance's state).

We probably need some combination of the following code to recover from this:

  • Sled agent has to enumerate the running Propolis zones on the system during startup
    • Each Propolis needs to be registered into the instance manager
    • Sled agent also needs to interrogate each Propolis to get the instance's current state
  • Sled agent must find out (either by asking or being told by Nexus) what instance state Nexus last received from the sled & must reconcile this with the current Propolis state, so that subsequent state updates will bear the correct generation numbers
    • Another possible approach is to cut Propolis in on generation numbering so that, if the VM survives, its state generation survives too

Initial triage: Marking this for MVP since, unless I've overlooked something, a sled agent reboot may functionally wedge a sled: sled agent loses track of all the instances, but they'll still hang around consuming the sled's resources. (We might not try to provision anything there, depending on how Nexus does bookkeeping for the zombie instances, but either way this, ah, suboptimal for capacity management purposes.)

Again, I could totally be overlooking something here, and I'd be happy if I were, but wanted to file something to make sure this isn't lost if this is indeed a problem--sled agent seems especially load-bearing in this respect and we could stand to try to mitigate this if it is.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing sled agent startup, Propolis zone discovery, and instance-manager registration, then inspect how Nexus state and generation numbers are reconciled after a restart. Confirm the observed teardown and recovery behavior for surviving VMs. Done means a restarted sled agent can rediscover running zones, restore their state, and report consistent updates to Nexus.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.