oxidecomputer / oxidecomputer/omicron
sled-agent's call to Propolis' `instance_ensure` failing can place a VMM that never started in `Failed` when it could be `SagaUnwound`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
When sled-agent calls instance_ensure in a way that fails to install a running Propolis zone, it calls fail_vmm_and_terminate, here:
https://github.com/oxidecomputer/omicron/blob/ff546dc31ec35a9d336d66f00cb60707bfa4cd6d/sled-agent/src/instance.rs#L2213-L2243
fail_vmm_and_terminate which sets the state to Failed and then calls terminate().
https://github.com/oxidecomputer/omicron/blob/ff546dc31ec35a9d336d66f00cb60707bfa4cd6d/sled-agent/src/instance.rs#L2547-L2552
this results in the InstanceRunner main loop exiting, which publishes the Failed state to Nexus:
You can see all of this in this log @jmpesp sent me:
16:58:33.898Z INFO SledAgent (InstanceManager): result of instance_ensure call is Err(Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "66a29faa-636f-403b-b023-f7fb76e34503", "content-length": "124", "date": "Fri, 29 May 2026 16:58:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "66a29faa-636f-403b-b023-f7fb76e34503" })
file = sled-agent/src/instance.rs:1227
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.898Z ERRO SledAgent (InstanceManager): failed to create Propolis VM
error = Failure from Propolis Client: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "66a29faa-636f-403b-b023-f7fb76e34503", "content-length": "124", "date": "Fri, 29 May 2026 16:58:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "66a29faa-636f-403b-b023-f7fb76e34503" }
file = sled-agent/src/instance.rs:2201
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): fail vmm and terminate
file = sled-agent/src/instance.rs:2579
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): force_state_to_failed
file = sled-agent/src/instance.rs:2581
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): terminate
file = sled-agent/src/instance.rs:2583
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): instance runner exited main loop
file = sled-agent/src/instance.rs:842
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
file = sled-agent/src/instance.rs:936
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
state = SledVmmState { vmm_state: VmmRuntimeState { state: Failed, generation: Generation(3), time_updated: 2026-05-29T16:58:33.900068983Z }, migration_in: None, migration_out: None }
16:58:38.634Z INFO SledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running
file = illumos-utils/src/zone.rs:492
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
zone = oxz_propolis-server_12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:38.634Z INFO SledAgent (InstanceManager): Stopped and uninstalled zone
file = illumos-utils/src/running_zone.rs:638
instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
zone = oxz_propolis-server_12988a63-f5ec-4fce-aab8-de36195d2f97
Rather than bubbling up an error to Nexus, the sled-agent instead returns a successful HTTP response with an instance state that happens to be Failed. This results in the instance-start saga succeeding, even though starting the instance on the sled failed. It probably shouldn't do that. We should probably instead be returning an error from that call in the case where we never actually managed to start a Propolis zone.
That way, the instance_start saga would instead unwind, put the instance in the saga_unwound state, and actually bubble up the error to the caller.
The current behavior is a bit of a bummer, for two reasons:
- Whoever called
instance_startwill receive a response indicating the instance started okay, but checking on the instance's state will say that it'sFailed. This is a shame --- it would be nicer for the caller to get an error response saying the instance failed to start here. - The VMM will be
Failedrather thanSagaUnwound, requiring that an instance update saga runs to clean up the failed VMM before the instance can be started again. If we had placed it inSagaUnwoundinstead, it could be started immediately, because the instance never actually started running on a sled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in sled-agent/src/instance.rs at the instance_ensure failure path around lines 2213-2243, then trace fail_vmm_and_terminate and the InstanceRunner main loop around lines 820-847. Determine how a failure before a Propolis zone starts should propagate to Nexus. Done means instance_start receives an error, the instance reaches SagaUnwound rather than Failed, and the existing cleanup behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100