oxidecomputer / oxidecomputer/propolis

instance ensure could be fully asynchronous

Open
#471 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

instance_ensure_common blocks on initialization of the VM controller, which includes steps that synchronously allocate all guest resources and set up all guest entities: https://github.com/oxidecomputer/propolis/blob/fbd701c0a54f25208712b0b3b2dc7931c875d347/bin/propolis-server/src/lib/server.rs#L486-L522

If this takes a long time (e.g. because Crucible activation takes a while, because it takes a while to set up guest memory, etc.), the client might time out even though the VM will eventually be able to start. This can cause issues like https://github.com/oxidecomputer/omicron/issues/3417, which appears to be a case where Nexus gave up on a sled agent call to start an instance that took longer than Nexus wanted.

We should certainly optimize VM start times as much as possible, but it would also be great to deal with the long tail of slow instance startup cases by making startup fully asynchronous:

  • Calling instance_ensure creates a VM controller but then returns immediately
  • The VM controller sets up entities and starts vCPUs asynchronously
    • Calls to instance_get that occur while this is happening report that the VM is "Starting"
  • Requests to change the VM state while it's being created are handled appropriately
    • Calls to stop the VM abort creation and destroy the controller
    • Other requests (e.g. reboot, migrate) fail with a "wrong state for operation" error status

Triage: marking as Unscheduled for now; many of these issues should be mitigated in practice by using the reservoir to allocate guest memory, and there are probably still opportunities to improve startup times further that we might want to pursue first.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with instance_ensure_common and the linked VM-controller initialization in bin/propolis-server/src/lib/server.rs#L486-L522. Review how instance_get and state-changing requests behave during creation, then define completion against the requested behavior: immediate ensure return, a "Starting" state, stop aborting creation, and other incompatible operations returning a wrong-state error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.