oxidecomputer / oxidecomputer/omicron
host OS update execution could pay more attention to slot numbers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
During the mess that was #10392, a sled had R19.2 in both phase 1 slots and R18 in both phase 2 slots. This is decidedly invalid and should never happen. But the next question is: why didn't the system automatically fix the divergence? The planner did notice the problem and configured an MGS update (some pieces redacted here):
{
"by_baseboard": [
{
"baseboard_id": { ... },
"sp_type": "sled",
"slot_id": 17,
"details": {
"component": "host_phase1",
"expected_active_phase_1_slot": "A",
"expected_boot_disk": "A",
"expected_active_phase_1_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
"expected_active_phase_2_hash": "ce39ff088ff5b95c580e4d84d8d1071d4d472f7c4d7df5f136fbfe2ad00b3241",
"expected_inactive_phase_1_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
"expected_inactive_phase_2_hash": "9ef2303694dac488ac6e473961133eb34a187d7dd42c04eae3ffe6b376f5b24b",
"sled_agent_address": "..."
},
"artifact_hash": "27220e68e1024ef636c039f631aa7e40b96bbcd3044861acc48fc1626533fff2",
"artifact_version": "19.2.0-0.ci+gite4b75dde134"
}
]
}
The MGS update driver concluded that the update was completed without having to do anything because it noticed:
- the system booted from slot A (I think -- to-be-confirmed)
- slot A's phase 1 is the right hash
- therefore the system has already been updated to the right thing
All of that is true. The problem is that then phase 2 was rewritten incorrectly to both slots. The planner is trying to repair that (admittedly, not fully grokking what's going on here). If the executor's end condition verified not just the above, but also that the system had booted out of the slot implied by the planner's request, we believe the system would have corrected itself. More specifically: if it had noticed that the system had booted from slot A, but the planner was giving it a precondition that the expected active slot before the update was A, then the executor could infer that it should be booting out of B and not consider the update done until all of the above is true for slot B. Then it would proceed to do the update for slot B anyway (writing phase 1 to slot B, then bouncing the host). We believe this would have repaired the problem.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the MGS update driver's completion check and the planner's expected_active_phase_1_slot precondition handling. Use the R19.2/R18 slot-divergence scenario described here to verify how the expected slot is inferred. Done means the executor does not declare success until the intended slot satisfies the expected hashes and the host has been bounced as required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100