oxidecomputer / oxidecomputer/omicron

sled-agent: `RunningZone::stop()` can't be retried on failure

Open
#7,881 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

RunningZone::stop() returns a Result<(), String>, but sled-agent doesn't handle (potential) errors from this very well. Current callers do one of the following:

I assume in practice we see errors from this function approximately never, given we haven't seen any sled-agent panics due to this. But this ultimately lands in Zones::halt_and_remove(), which can fail at three different points (halting the zone, uninstalling the zone, deleting the zone config).

I'm doing some restructuring work that would allow at least the services case to retry on failure, but I think the current implementation of RunningZone::stop() makes that impossible: on first call, it calls self.id.take(), so regardless of the success or failure of the subsequent halt_and_remove, any future calls to stop() will return Ok(()) without actually attempting to shut down the zone. It looks like Zones::halt_and_remove tries to be resumable on failure; could we just drop the self.id.take() altogether from RunningZone::stop()? Or perhaps only clear self.id if we succeeded in shutting down the zone? Maybe "shut down and remove" should be split, and we should clear self.id if we've successfully halted the zone even if we haven't uninstalled it?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with RunningZone::stop() in illumos-utils/src/running_zone.rs and follow Zones::halt_and_remove() in illumos-utils/src/zone.rs. Then inspect the sled-agent callers in services.rs, probe_manager.rs, and instance.rs to understand their error handling. Done means a failed stop can be retried without being reported as a successful no-op, with the zone lifecycle semantics made consistent for all callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.