oxidecomputer / oxidecomputer/omicron
sled-agent: `RunningZone::stop()` can't be retried on failure
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
RunningZone::stop() returns a Result<(), String>, but sled-agent doesn't handle (potential) errors from this very well. Current callers do one of the following:
- Log the error but never retry (omicron services and probes)
- Ignore errors (switch zone)
- Panic on error (!) (instances)
I assume in practice we see errors from this function approximately never, given we haven't seen any sled-agent panics due to this. But this ultimately lands in Zones::halt_and_remove(), which can fail at three different points (halting the zone, uninstalling the zone, deleting the zone config).
I'm doing some restructuring work that would allow at least the services case to retry on failure, but I think the current implementation of RunningZone::stop() makes that impossible: on first call, it calls self.id.take(), so regardless of the success or failure of the subsequent halt_and_remove, any future calls to stop() will return Ok(()) without actually attempting to shut down the zone. It looks like Zones::halt_and_remove tries to be resumable on failure; could we just drop the self.id.take() altogether from RunningZone::stop()? Or perhaps only clear self.id if we succeeded in shutting down the zone? Maybe "shut down and remove" should be split, and we should clear self.id if we've successfully halted the zone even if we haven't uninstalled it?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RunningZone::stop() in illumos-utils/src/running_zone.rs and follow Zones::halt_and_remove() in illumos-utils/src/zone.rs. Then inspect the sled-agent callers in services.rs, probe_manager.rs, and instance.rs to understand their error handling. Done means a failed stop can be retried without being reported as a successful no-op, with the zone lifecycle semantics made consistent for all callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100