oxidecomputer / oxidecomputer/omicron

host OS update execution could pay more attention to slot numbers

Open
#10,395 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

During the mess that was #10392, a sled had R19.2 in both phase 1 slots and R18 in both phase 2 slots. This is decidedly invalid and should never happen. But the next question is: why didn't the system automatically fix the divergence? The planner did notice the problem and configured an MGS update (some pieces redacted here):

{
  "by_baseboard": [
    {
      "baseboard_id": { ... },
      "sp_type": "sled",
      "slot_id": 17,
      "details": {
        "component": "host_phase1",
        "expected_active_phase_1_slot": "A",
        "expected_boot_disk": "A",
        "expected_active_phase_1_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
        "expected_active_phase_2_hash": "ce39ff088ff5b95c580e4d84d8d1071d4d472f7c4d7df5f136fbfe2ad00b3241",
        "expected_inactive_phase_1_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
        "expected_inactive_phase_2_hash": "9ef2303694dac488ac6e473961133eb34a187d7dd42c04eae3ffe6b376f5b24b",
        "sled_agent_address": "..."
      },
      "artifact_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
      "artifact_version": "19.2.0-0.ci+gite4b75dde134"
    }
  ]
}

The MGS update driver concluded that the update was completed without having to do anything because it noticed:

  • the system booted from slot A (I think -- to-be-confirmed)
  • slot A's phase 1 is the right hash
  • therefore the system has already been updated to the right thing

All of that is true. The problem is that then phase 2 was rewritten incorrectly to both slots. The planner is trying to repair that (admittedly, not fully grokking what's going on here). If the executor's end condition verified not just the above, but also that the system had booted out of the slot implied by the planner's request, we believe the system would have corrected itself. More specifically: if it had noticed that the system had booted from slot A, but the planner was giving it a precondition that the expected active slot before the update was A, then the executor could infer that it should be booting out of B and not consider the update done until all of the above is true for slot B. Then it would proceed to do the update for slot B anyway (writing phase 1 to slot B, then bouncing the host). We believe this would have repaired the problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the MGS update driver's completion check and the planner's expected_active_phase_1_slot precondition handling. Use the R19.2/R18 slot-divergence scenario described here to verify how the expected slot is inferred. Done means the executor does not declare success until the intended slot satisfies the expected hashes and the host has been bounced as required.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.