oxidecomputer / oxidecomputer/propolis
instance ensure could be fully asynchronous
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 270
- Forks
- 42
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 6
Description
instance_ensure_common blocks on initialization of the VM controller, which includes steps that synchronously allocate all guest resources and set up all guest entities: https://github.com/oxidecomputer/propolis/blob/fbd701c0a54f25208712b0b3b2dc7931c875d347/bin/propolis-server/src/lib/server.rs#L486-L522
If this takes a long time (e.g. because Crucible activation takes a while, because it takes a while to set up guest memory, etc.), the client might time out even though the VM will eventually be able to start. This can cause issues like https://github.com/oxidecomputer/omicron/issues/3417, which appears to be a case where Nexus gave up on a sled agent call to start an instance that took longer than Nexus wanted.
We should certainly optimize VM start times as much as possible, but it would also be great to deal with the long tail of slow instance startup cases by making startup fully asynchronous:
- Calling
instance_ensurecreates a VM controller but then returns immediately - The VM controller sets up entities and starts vCPUs asynchronously
- Calls to
instance_getthat occur while this is happening report that the VM is "Starting"
- Calls to
- Requests to change the VM state while it's being created are handled appropriately
- Calls to stop the VM abort creation and destroy the controller
- Other requests (e.g. reboot, migrate) fail with a "wrong state for operation" error status
Triage: marking as Unscheduled for now; many of these issues should be mitigated in practice by using the reservoir to allocate guest memory, and there are probably still opportunities to improve startup times further that we might want to pursue first.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with instance_ensure_common and the linked VM-controller initialization in bin/propolis-server/src/lib/server.rs#L486-L522. Review how instance_get and state-changing requests behave during creation, then define completion against the requested behavior: immediate ensure return, a "Starting" state, stop aborting creation, and other incompatible operations returning a wrong-state error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100