oxidecomputer / oxidecomputer/propolis

over-eager to destroy VM when device start() fails

Open
#1,115 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

device start is a fallible operation! we try to start devices in propolis-server:

https://github.com/oxidecomputer/propolis/blob/fcf37aefdc1d3bcce3cbf7e2b984e05ccfe6b553/bin/propolis-server/src/lib/vm/state_driver.rs#L596-L609

if a device fails to start (or a block backend, below, fails to start), we'll then return VmStartOutcome::Failed and eventually eventually get to HandleEventOutcome::Exit. then we set_rundown() and drop the StateDriver, eventually getting through

https://github.com/oxidecomputer/propolis/blob/fcf37aefdc1d3bcce3cbf7e2b984e05ccfe6b553/bin/propolis-server/src/lib/vm/objects.rs#L420-L433

at this point we'll have dropped the Machine, uninstalled the guest's memory (and MSI-X handle!) from devices, and dropped everything. some devices will have been started, some will not yet have started. devices that have started will have some parts dropped by Propolis, but spawned threads and spawned tasks may not be cancelled, stopped, joined on, etc.

#1110 is, in part, because the vsock poller expects that once it has memory, memory only goes away after the device is paused and the poller thread is told to exit. we probably should expect that Propolis embedders follow the valid state transitions described by Indicator, and that if a device is started it must be paused before dropping the Machine, that a device is halted when dropped, etc.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the failed-start path in bin/propolis-server/src/lib/vm/state_driver.rs, including set_rundown(), then inspect Machine cleanup in bin/propolis-server/src/lib/vm/objects.rs. Read the valid lifecycle transitions in lib/propolis/src/lifecycle.rs and compare them with the device-start failure path. Done means started devices are paused or halted and their spawned work is stopped before the Machine and guest memory are dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.