oxidecomputer / oxidecomputer/omicron

A snapshot-create saga unwind does not clean up a partially successful snapshot

Open
#10,735 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

One of the sharpest edges of sagas strikes again! Crucible's snapshot operation is not atomic: snapshots require all three downstairs to answer:

            IOop::Flush {
                snapshot_details, ..
            } => {
                let n = if deactivate || snapshot_details.is_some() {
                    3
                } else {
                    2
                };
                okay == n
            }

but If any of the downstairs return an error for a snapshot, then nothing in Crucible will delete the other successful ones.

Nexus should be deleting those, but due to the property of sagas where an unwind is not run for forward actions that do not complete, and due to the forward action and unwind action being part of the same node, this does not occur:

    SEND_SNAPSHOT_REQUEST_TO_SLED_AGENT -> "snapshot_request_to_sled_agent" {
        + ssc_send_snapshot_request_to_sled_agent
        - ssc_send_snapshot_request_to_sled_agent_undo
    }

omdb db validate validate-region-snapshots shows that one of the three downstairs for two different snapshots did not work, and the other two did not get cleaned up:

 dataset_id                           | region_id                            | snapshot_id                          | dataset_addr                   | error                                                     
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------+-----------------------------------------------------------
 5ec8bf72-c69c-4d8a-b8eb-c14b15be2953 | 017eb81c-de6e-4edd-89c7-1e36ccb498c7 | e660819a-723e-4e13-994b-33b3f8a1ce8e | [fd00:1122:3344:127::23]:32345 | Nexus does not know about this snapshot!                  
 fd58cfe0-4b02-42c1-99d2-bbde5d1ae9d8 | 7a8ca856-1d0e-4b28-9cf5-7078ec5fa4a8 | 004f5980-9763-4dfc-91e8-089d7342d2a8 | [fd00:1122:3344:129::28]:32345 | Nexus does not know about this snapshot!                  
 1876cdcf-b2e7-4b79-ad2e-67df716e1860 | 5cbf48f1-5081-40f6-873d-8125f461619a | e660819a-723e-4e13-994b-33b3f8a1ce8e | [fd00:1122:3344:10a::8]:32345  | Nexus does not know about this snapshot!                  
 92d3e4e9-0768-4772-83c1-23cce52190e9 | f0e488b3-b874-40eb-a181-0a789fc4b8a6 | 004f5980-9763-4dfc-91e8-089d7342d2a8 | [fd00:1122:3344:105::6]:32345  | Nexus does not know about this snapshot!

The problem is that this causes all delete related operations to not work until those region snapshots are manually deleted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the snapshot handling in upstairs/src/downstairs.rs and the SEND_SNAPSHOT_REQUEST_TO_SLED_AGENT saga actions, including their forward and undo functions. Trace the partial-success path and the omdb db validate validate-region-snapshots output; done means a failed snapshot saga removes any successful downstairs snapshots so orphaned region snapshots no longer block deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.