oxidecomputer / oxidecomputer/omicron

Planner defers all zone updates when any zone is unhealthy

Open
#10,323 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

We have observed two customer-driven updates where the zone upgrade portion does not begin due to a failed U.2 drive.

The planner correctly determines that the zpool is unavailable and its associated crucible is down:

a93cdff6-bf17-420f-ae9d-9b91a0f3fb2f crucible        c262cbe0-60f6-4881-a39a-11b5522d3e26 zone's transient root dataset is not available: oxp_f27830df-8e50-4c3c-888b-056488ff2bb8/crypt/zone/oxz_crucible_c262cbe0-60f6-4881-a39a-11b5522d3e26

The SP and host OS updates are performed, but all of the zone upgrades are deferred until the problem disk and its associated zone are expunged:

                  |18.0.0-0.ci+gitd211c40e1e5 |19.2.0-0.ci+gite4b75dde134 |error 
------------------+---------------------------+---------------------------+------
RoT bootloader    |0                          |28                         |0     
RoT               |0                          |28                         |0     
SP                |0                          |28                         |0     
Host OS (phase 1) |0                          |24                         |0     
Host OS (phase 2) |0                          |24                         |0     
Zone              |282                        |0                          |1     

It would be preferable for the system to proceed with the zone updates that are not impacted by the hardware failure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/reconfigurator/planning/src/planner.rs at lines 1644-1660, where zone upgrades are deferred when a zone is unhealthy. Trace how the failed U.2 drive and unavailable crucible affect the planner's zone decisions. Done means unaffected zones proceed with their upgrades while the impacted zone remains deferred.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.