oxidecomputer / oxidecomputer/omicron
Failures to delete freed regions can cascade
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Apologies if this is a duplicate issue. I thought we had one already but I could not find it.
When a disk delete occurs, once of the steps in the saga is to scan for freed regions and perform deletes on them. The regions to be freed looks to be a global scan (as opposed to those directly freed by the delete being processed). This means that under the following circumstances:
- Disk is deleted
- Region is found to be freed
- Delete is attempted, but fails due to a downstairs sled being unresponsive
The disk delete saga will be stuck in a retry loop attempting to contact the downstairs. From this point on, any subsequent disk delete will see the same freed region in its scan, and will therefore also get stuck in a retry loop attempting to delete it.
I'm not sure the correct solution to this, but it seems like freed region cleanup should possibly be decoupled from the disk delete and moved to a background task. I do not know the consequences of this though.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the disk delete saga’s global freed-region scan and the retry path when a downstairs sled is unresponsive. Determine whether cleanup can be decoupled into a background task without cascading failures, and verify that later disk deletes are not blocked by a failed region cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100