oxidecomputer / oxidecomputer/omicron

Consider persisting the history of planning inputs along with sitreps

Open
#11,309 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

fault-management
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

One of the primary use cases for the FM simulator is to help debug when the planning phase (i.e. our collection of diagnosis engines, SitrepBuilder and associated machinery) have done the wrong thing in production. The idea is that the operator would export a support bundle containing enough data for us to reproduce what happened.

The trouble is that fm_analysis is a background task running across multiple Nexus instances at effectively arbitrary intervals, each observing a different snapshot of the system as they run, and that snapshot isn't persisted immutably anywhere. It is processed to form the Input for the analysis, and then further summarized into an InputReport, which is better than nothing—that is, it's human readable—but isn't sufficient to replay what happened in the simulator.

What we're most likely to do for the initial MVP of the FM simulator is just allow operators to dump the current state. That would look something like:

#[derive(Serialize, Deserialize)]
pub struct UnstableFmState {
    /// The planning input exactly as production would analyze it.
    pub input: Input,
    /// What `build` consumed to decide carry-forward; the sim needs them as state.
    pub marked_alert_requests: BTreeSet<AlertUuid>,
    pub marked_support_bundle_requests: BTreeSet<SupportBundleUuid>,
    pub unmarked_seen_ereports: BTreeSet<EreportId>,
    /// Context the planner doesn't currently read but the sim's world needs.
    pub nexus_zones: BTreeMap<OmicronZoneUuid, NexusZoneState>,
}

// in analysis_input
impl Builder {
    pub fn build(self) -> Result<(Input, Report), InvalidInputs>   // unchanged
    pub fn build_unstable_state(self, nexus_zones: ...) -> Result<UnstableFmState, InvalidInputs>
}

We'd serialize this UnstableFmState and include it in a support bundle, which would at least be useful to let us see what weird states FM can get itself stuck in, and let us prove to ourselves that we can get ourselves unstuck from those states. This issue proposes extending this mechanism a bit: every time fm_analysis runs, we'd also serialize UnstableFmState as a compressed JSON blob in the database and store it alongside its corresponding sitrep. The hope is that, by including this full history in a support bundle rather than just the current state, we could bisect through that history to see exactly where things went wrong.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the fm_analysis background task and analysis_input::Builder, then inspect how sitreps are persisted and how support bundles are assembled. Determine the database and export points needed for UnstableFmState history. Done means each run's compressed state is stored with its sitrep and the history can be included for simulator replay and bisecting.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.