oxidecomputer / oxidecomputer/omicron

sled-agent fails to clean up old zone-bundler snapshots on startup

Open
#7,746 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Following up on #7743, @smklein and I were trying to figure out why dogfood had a handful of very old (late 2023) zone-archives snapshots. Early in its life sled-agent creates a ZoneBundler, and one of the first things it does is try to find and destroy old snapshots in initialize_zfs_resources: https://github.com/oxidecomputer/omicron/blob/078678f181bb5581124d740dc929a69c66989b6f/sled-agent/src/zone_bundle.rs#L70-L110.

There are a handful of bugs here:

  1. The boolean conditional on line 75 should be &&, not ||. As written, it is never true, so we never proceed to checking for the zone-bundle-specific property. (In hindsight, this is saving us!)
  2. On line 87, we call get_oxide_value(_, ZONE_BUNDLE_ZFS_PROPERTY_NAME). ZONE_BUNDLE_ZFS_PROPERTY_NAME is "oxide:for-zone-bundle"; however, get_oxide_value then prepends "oxide:" itself, which means we're erroneously querying for oxide:oxide:for-zone-bundle.
  3. Inside get_oxide_value, we call get_values(..., Some(PropertySource::Local)). When get_values is given a non-None property source, it inserts -s $source in the command line args, but in the wrong place: it needs to come before all_names. As written, the zfs get invocation will fail.
  4. Back in initialize_zfs_resources: on line 88 we panic on any failure from the get_oxide_value() call, which at the moment is guaranteed to fail due to items 2 and 3. This could also fail for any spurious reason. Probably we should log a warning and return false here instead (which would skip destroying the dataset, but that seems okay)?
  5. On line 98 we assert that the value we read back is true, which could also induce a panic. Can we log a warning and return false here too?

I believe fixing this set all together would let dogfood clean up these old snapshots as intended. But we've also discussed removing zone bundles altogether now that support bundles are coming along. If we do that we may want to manually clean up these old snapshots (and possible check customer systems for them?).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent/src/zone_bundle.rs lines 70-110 and the related helpers in illumos-utils/src/zfs.rs, especially get_oxide_value and get_values. Verify the property lookup, zfs argument ordering, and failure handling so startup cleanup does not panic; done means old zone-bundler snapshots can be evaluated and safely cleaned up, unless zone bundles are removed instead.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.