oxidecomputer / oxidecomputer/omicron
sled-agent fails to clean up old zone-bundler snapshots on startup
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Following up on #7743, @smklein and I were trying to figure out why dogfood had a handful of very old (late 2023) zone-archives snapshots. Early in its life sled-agent creates a ZoneBundler, and one of the first things it does is try to find and destroy old snapshots in initialize_zfs_resources: https://github.com/oxidecomputer/omicron/blob/078678f181bb5581124d740dc929a69c66989b6f/sled-agent/src/zone_bundle.rs#L70-L110.
There are a handful of bugs here:
- The boolean conditional on line 75 should be
&&, not||. As written, it is never true, so we never proceed to checking for the zone-bundle-specific property. (In hindsight, this is saving us!) - On line 87, we call
get_oxide_value(_, ZONE_BUNDLE_ZFS_PROPERTY_NAME).ZONE_BUNDLE_ZFS_PROPERTY_NAMEis"oxide:for-zone-bundle"; however,get_oxide_valuethen prepends"oxide:"itself, which means we're erroneously querying foroxide:oxide:for-zone-bundle. - Inside
get_oxide_value, we callget_values(..., Some(PropertySource::Local)). Whenget_valuesis given a non-Noneproperty source, it inserts-s $sourcein the command line args, but in the wrong place: it needs to come beforeall_names. As written, thezfs getinvocation will fail. - Back in
initialize_zfs_resources: on line 88 we panic on any failure from theget_oxide_value()call, which at the moment is guaranteed to fail due to items 2 and 3. This could also fail for any spurious reason. Probably we should log a warning and return false here instead (which would skip destroying the dataset, but that seems okay)? - On line 98 we
assertthat the value we read back istrue, which could also induce a panic. Can we log a warning and return false here too?
I believe fixing this set all together would let dogfood clean up these old snapshots as intended. But we've also discussed removing zone bundles altogether now that support bundles are coming along. If we do that we may want to manually clean up these old snapshots (and possible check customer systems for them?).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sled-agent/src/zone_bundle.rs lines 70-110 and the related helpers in illumos-utils/src/zfs.rs, especially get_oxide_value and get_values. Verify the property lookup, zfs argument ordering, and failure handling so startup cleanup does not panic; done means old zone-bundler snapshots can be evaluated and safely cleaned up, unless zone bundles are removed instead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100