oxidecomputer / oxidecomputer/omicron

Failures to delete freed regions can cascade

Open
#8,010 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Apologies if this is a duplicate issue. I thought we had one already but I could not find it.

When a disk delete occurs, once of the steps in the saga is to scan for freed regions and perform deletes on them. The regions to be freed looks to be a global scan (as opposed to those directly freed by the delete being processed). This means that under the following circumstances:

  1. Disk is deleted
  2. Region is found to be freed
  3. Delete is attempted, but fails due to a downstairs sled being unresponsive

The disk delete saga will be stuck in a retry loop attempting to contact the downstairs. From this point on, any subsequent disk delete will see the same freed region in its scan, and will therefore also get stuck in a retry loop attempting to delete it.

I'm not sure the correct solution to this, but it seems like freed region cleanup should possibly be decoupled from the disk delete and moved to a background task. I do not know the consequences of this though.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the disk delete saga’s global freed-region scan and the retry path when a downstairs sled is unresponsive. Determine whether cleanup can be decoupled into a background task without cascading failures, and verify that later disk deletes are not blocked by a failed region cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.