oxidecomputer / oxidecomputer/omicron

VMM landed on sled with an eventual 0 reservoir

Open
#11,130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

While looking into reservoir issues we noticed a sled for which Nexus believe the reservoir was 0, but also contained a propolis zone. Based on logs it looks like the order of events were that the sled was rebooted (as part of an automated updated), and when it came back up it started to allocate the reservoir (pre-timesync). We confirmed that prior to this update the reservoir has been set correctly and so Nexus knew the size that it "should be" upon a successful allocation.

A VMM PUT was issued to the sled for an instance that had a local disk allocated on it. This means that this is the only sled that the VM could be placed after having been shutdown from the reboot. We see that post timesync, but pre-reservoir failure that we begin to install the propolis zone.

2026-08-12T21:24:30.36023045Z -> Installing Omicron zone: oxz_propolis-server_457cbf06-5c6a-4287-927d-101ce8eb0c30

But then a few minutes later we fail to resize the reservoir:

2026-08-12T21:27:06.348569867Z -> sled agent reservoir failure

This in turns reports a 0 to Nexus for the reservoir size.

Currently this is a hypothesis based on the the sled-agent log, but needs testing to confirm it. Log file is available at:
/staff/rack3/BRM42220015/2026-08-21/oxide-sled-agent-default.log.1786581134

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sled-agent log at /staff/rack3/BRM42220015/2026-08-21/oxide-sled-agent-default.log.1786581134 and trace the VMM PUT, propolis zone installation, and reservoir resize events described in the issue. Confirm whether the ordering can make Nexus observe a zero reservoir, then add or run a regression test showing the corrected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.