oxidecomputer / oxidecomputer/omicron
Perform disk delete via background tasks
@jmpesp is already working on this.
Since Sep 14, 2026.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Disk, snapshot, and image deletes have historically been the source of a lot of problems, especially when hardware is flaky. The whole associated delete saga had to complete before any of the endpoints would return an OK message, and they would hang if the saga was in a retry loop, often until an expungement occurred that would unblock said retry loop. Even worse was if the "delete freed regions" step was the one that was blocked, as that would hang any and every delete of a volume until resolved.
In accordance with the guidance in RFD 704, we're looking at replacing problematic sagas with one or more background tasks. The disk delete saga was this author's first target: the whole saga was comprised of nodes that only have forward actions, so it was an ideal candidate for this conversion, and sagas that are like this don't make much sense in the first place.
Note that using background tasks for the actual resource clean-up will cause a behaviour change: today, the disk / snapshot / image delete API calls do not return until all the associated resources are cleaned up (in the case of Crucible backed disks, cleaned up as much as possible). After this change is made is merged, it will be possible to delete a disk, quickly create another one, and temporarily see an INSUFFICIENT_STORAGE error, because the background task that would have cleaned up either the regions or the local storage allocation hasn't fired yet. It would be incorrect to spin in the disk / snapshot / image delete endpoints waiting for something, as that would lead us into the same problems we had before.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.