oxidecomputer / oxidecomputer/omicron
VMM landed on sled with an eventual 0 reservoir
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
While looking into reservoir issues we noticed a sled for which Nexus believe the reservoir was 0, but also contained a propolis zone. Based on logs it looks like the order of events were that the sled was rebooted (as part of an automated updated), and when it came back up it started to allocate the reservoir (pre-timesync). We confirmed that prior to this update the reservoir has been set correctly and so Nexus knew the size that it "should be" upon a successful allocation.
A VMM PUT was issued to the sled for an instance that had a local disk allocated on it. This means that this is the only sled that the VM could be placed after having been shutdown from the reboot. We see that post timesync, but pre-reservoir failure that we begin to install the propolis zone.
2026-08-12T21:24:30.36023045Z -> Installing Omicron zone: oxz_propolis-server_457cbf06-5c6a-4287-927d-101ce8eb0c30
But then a few minutes later we fail to resize the reservoir:
2026-08-12T21:27:06.348569867Z -> sled agent reservoir failure
This in turns reports a 0 to Nexus for the reservoir size.
Currently this is a hypothesis based on the the sled-agent log, but needs testing to confirm it. Log file is available at:
/staff/rack3/BRM42220015/2026-08-21/oxide-sled-agent-default.log.1786581134
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the sled-agent log at /staff/rack3/BRM42220015/2026-08-21/oxide-sled-agent-default.log.1786581134 and trace the VMM PUT, propolis zone installation, and reservoir resize events described in the issue. Confirm whether the ordering can make Nexus observe a zero reservoir, then add or run a regression test showing the corrected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100