oxidecomputer / oxidecomputer/hubris
SP should not auto-reboot host in response to a host-reported boot failure
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
(Fallout from #1613)
Currently, if the host reports a boot failure over the IPCC link, we respond by recording the information and resuming normal business. When the host immediately follows that up with a reboot request, we dutifully reboot the host.
Because we haven't taken any additional actions to fix the boot failure (by, for instance, flipping the host flash mux), this will probably always produce a reboot loop.
While this sort of reboot loop is likely not destructive, it's distracting: the machine cycles, the logs/ringbufs get overwritten, power is wasted, etc. I think after a boot failure like this, we should probably not attempt to boot the host until we have reason to believe the failure has been repaired.
The SP itself doesn't have sufficient context to know how to "repair" such a failure. If the failure was hit while attempting a recover image boot through Wicket, for instance, we specifically do not want to do an automatic slot fallback. If we hit it during a production software upgrade, we might, depending on circumstances, want to do a slot fallback. The right answer in basically all cases appears to be: escalate to the control plane, where context is more easily available.
So, I think we should stop rebooting the host after a boot failure, period, and wait for messages over the network. The boot failure is stored in a place the control plane can get to it (in the control-plane-agent). If we had a way of proactively sounding an alarm, we could do that, but for now it'd have to be polled.
Concretely, I discussed this briefly with @wesolows and the simplest thing appears to be:
- Honor reboot requests from the host normally, except
- If we get a host boot failure message, set a flag that causes the next reboot request to be interpreted as "power down and intervene."
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the host boot-failure handling and subsequent reboot-request path described in the issue. Identify where state can record the failure and where the next reboot request is interpreted; done means that request powers down for intervention rather than rebooting automatically, while ordinary reboot requests remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100