oxidecomputer / oxidecomputer/omicron
A snapshot-create saga unwind does not clean up a partially successful snapshot
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
One of the sharpest edges of sagas strikes again! Crucible's snapshot operation is not atomic: snapshots require all three downstairs to answer:
IOop::Flush {
snapshot_details, ..
} => {
let n = if deactivate || snapshot_details.is_some() {
3
} else {
2
};
okay == n
}
but If any of the downstairs return an error for a snapshot, then nothing in Crucible will delete the other successful ones.
Nexus should be deleting those, but due to the property of sagas where an unwind is not run for forward actions that do not complete, and due to the forward action and unwind action being part of the same node, this does not occur:
SEND_SNAPSHOT_REQUEST_TO_SLED_AGENT -> "snapshot_request_to_sled_agent" {
+ ssc_send_snapshot_request_to_sled_agent
- ssc_send_snapshot_request_to_sled_agent_undo
}
omdb db validate validate-region-snapshots shows that one of the three downstairs for two different snapshots did not work, and the other two did not get cleaned up:
dataset_id | region_id | snapshot_id | dataset_addr | error
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------+-----------------------------------------------------------
5ec8bf72-c69c-4d8a-b8eb-c14b15be2953 | 017eb81c-de6e-4edd-89c7-1e36ccb498c7 | e660819a-723e-4e13-994b-33b3f8a1ce8e | [fd00:1122:3344:127::23]:32345 | Nexus does not know about this snapshot!
fd58cfe0-4b02-42c1-99d2-bbde5d1ae9d8 | 7a8ca856-1d0e-4b28-9cf5-7078ec5fa4a8 | 004f5980-9763-4dfc-91e8-089d7342d2a8 | [fd00:1122:3344:129::28]:32345 | Nexus does not know about this snapshot!
1876cdcf-b2e7-4b79-ad2e-67df716e1860 | 5cbf48f1-5081-40f6-873d-8125f461619a | e660819a-723e-4e13-994b-33b3f8a1ce8e | [fd00:1122:3344:10a::8]:32345 | Nexus does not know about this snapshot!
92d3e4e9-0768-4772-83c1-23cce52190e9 | f0e488b3-b874-40eb-a181-0a789fc4b8a6 | 004f5980-9763-4dfc-91e8-089d7342d2a8 | [fd00:1122:3344:105::6]:32345 | Nexus does not know about this snapshot!
The problem is that this causes all delete related operations to not work until those region snapshots are manually deleted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the snapshot handling in upstairs/src/downstairs.rs and the SEND_SNAPSHOT_REQUEST_TO_SLED_AGENT saga actions, including their forward and undo functions. Trace the partial-success path and the omdb db validate validate-region-snapshots output; done means a failed snapshot saga removes any successful downstairs snapshots so orphaned region snapshots no longer block deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100