oxidecomputer / oxidecomputer/omicron

non-global zone log files lost after sled agent restart

Open
#9,644 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I discovered this while testing #9226:

Losing logs after sled agent restart
If sled agent restarts, we also go through the above code path that wipes the datasets. But even before that, early during startup, sled agent halts and uninstalls all of zones that look like they were created by the control plane. This wipes their root filesystems. This happens long before almost anything else in sled agent is set up, including the DumpSetup subsystem, config reconciler, disk adoption, etc., so there's no opportunity to trigger archival at this point.

The flow is:

zoneadm(8) says about uninstall:

   uninstall [-F]

      Uninstall the specified zone from the system. Use this subcommand
      with caution.  It removes all of the files under the zonepath of
      the zone in question.  You can use the -F flag to force the action.

So this will wind up removing all of the zones' log files.


I'm not sure how best to fix this. We could halt the zones without uninstalling them, but we'd presumably later want to do the uninstall? But the later code assumes (because of this code) that no zones exist already so I'm not sure what that will confuse. (We don't want it to always uninstall the zones it finds -- they may have been ones started in this lifetime of sled agent.)

It's tempting to do the archival before all this, but this is so early in sled agent we're not even close to setting up the debug collector. In general, we also haven't adopted the disks yet and wouldn't know that we have a debug dataset in which to put the log files.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent/src/bin/sled-agent.rs and follow BootstrapAgentStartup::run into sled-agent/src/bootstrap/pre_server.rs, especially cleanup_all_old_global_state() and its zone removal flow. Read illumos-utils/src/zone.rs alongside the zoneadm uninstall behavior described here. Done means sled agent restarts no longer remove non-global zone log files while preserving the intended cleanup of stale state.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.