oxidecomputer / oxidecomputer/omicron

sled-agent's call to Propolis' `instance_ensure` failing can place a VMM that never started in `Failed` when it could be `SagaUnwound`

Open
#10,516 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

nexus Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When sled-agent calls instance_ensure in a way that fails to install a running Propolis zone, it calls fail_vmm_and_terminate, here:
https://github.com/oxidecomputer/omicron/blob/ff546dc31ec35a9d336d66f00cb60707bfa4cd6d/sled-agent/src/instance.rs#L2213-L2243

fail_vmm_and_terminate which sets the state to Failed and then calls terminate().
https://github.com/oxidecomputer/omicron/blob/ff546dc31ec35a9d336d66f00cb60707bfa4cd6d/sled-agent/src/instance.rs#L2547-L2552

this results in the InstanceRunner main loop exiting, which publishes the Failed state to Nexus:

https://github.com/oxidecomputer/omicron/blob/ff546dc31ec35a9d336d66f00cb60707bfa4cd6d/sled-agent/src/instance.rs#L820-L847

You can see all of this in this log @jmpesp sent me:

16:58:33.898Z INFO SledAgent (InstanceManager): result of instance_ensure call is Err(Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "66a29faa-636f-403b-b023-f7fb76e34503", "content-length": "124", "date": "Fri, 29 May 2026 16:58:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "66a29faa-636f-403b-b023-f7fb76e34503" })
    file = sled-agent/src/instance.rs:1227
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.898Z ERRO SledAgent (InstanceManager): failed to create Propolis VM
    error = Failure from Propolis Client: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "66a29faa-636f-403b-b023-f7fb76e34503", "content-length": "124", "date": "Fri, 29 May 2026 16:58:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "66a29faa-636f-403b-b023-f7fb76e34503" }
    file = sled-agent/src/instance.rs:2201
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): fail vmm and terminate
    file = sled-agent/src/instance.rs:2579
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): force_state_to_failed
    file = sled-agent/src/instance.rs:2581
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): terminate
    file = sled-agent/src/instance.rs:2583
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): instance runner exited main loop
    file = sled-agent/src/instance.rs:842
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:33.900Z INFO SledAgent (InstanceManager): Publishing instance state update to Nexus
    file = sled-agent/src/instance.rs:936
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
    state = SledVmmState { vmm_state: VmmRuntimeState { state: Failed, generation: Generation(3), time_updated: 2026-05-29T16:58:33.900068983Z }, migration_in: None, migration_out: None }
16:58:38.634Z INFO SledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running
    file = illumos-utils/src/zone.rs:492
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
    zone = oxz_propolis-server_12988a63-f5ec-4fce-aab8-de36195d2f97
16:58:38.634Z INFO SledAgent (InstanceManager): Stopped and uninstalled zone
    file = illumos-utils/src/running_zone.rs:638
    instance_id = 81e6555a-7ea9-45b5-8e13-83346c581d1e
    propolis_id = 12988a63-f5ec-4fce-aab8-de36195d2f97
    zone = oxz_propolis-server_12988a63-f5ec-4fce-aab8-de36195d2f97

Rather than bubbling up an error to Nexus, the sled-agent instead returns a successful HTTP response with an instance state that happens to be Failed. This results in the instance-start saga succeeding, even though starting the instance on the sled failed. It probably shouldn't do that. We should probably instead be returning an error from that call in the case where we never actually managed to start a Propolis zone.

That way, the instance_start saga would instead unwind, put the instance in the saga_unwound state, and actually bubble up the error to the caller.

The current behavior is a bit of a bummer, for two reasons:

  1. Whoever called instance_start will receive a response indicating the instance started okay, but checking on the instance's state will say that it's Failed. This is a shame --- it would be nicer for the caller to get an error response saying the instance failed to start here.
  2. The VMM will be Failed rather than SagaUnwound, requiring that an instance update saga runs to clean up the failed VMM before the instance can be started again. If we had placed it in SagaUnwound instead, it could be started immediately, because the instance never actually started running on a sled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sled-agent/src/instance.rs at the instance_ensure failure path around lines 2213-2243, then trace fail_vmm_and_terminate and the InstanceRunner main loop around lines 820-847. Determine how a failure before a Propolis zone starts should propagate to Nexus. Done means instance_start receives an error, the instance reaches SagaUnwound rather than Failed, and the existing cleanup behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.