oxidecomputer / oxidecomputer/hubris

SP should not auto-reboot host in response to a host-reported boot failure

Open
#1,614 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

(Fallout from #1613)

Currently, if the host reports a boot failure over the IPCC link, we respond by recording the information and resuming normal business. When the host immediately follows that up with a reboot request, we dutifully reboot the host.

Because we haven't taken any additional actions to fix the boot failure (by, for instance, flipping the host flash mux), this will probably always produce a reboot loop.

While this sort of reboot loop is likely not destructive, it's distracting: the machine cycles, the logs/ringbufs get overwritten, power is wasted, etc. I think after a boot failure like this, we should probably not attempt to boot the host until we have reason to believe the failure has been repaired.

The SP itself doesn't have sufficient context to know how to "repair" such a failure. If the failure was hit while attempting a recover image boot through Wicket, for instance, we specifically do not want to do an automatic slot fallback. If we hit it during a production software upgrade, we might, depending on circumstances, want to do a slot fallback. The right answer in basically all cases appears to be: escalate to the control plane, where context is more easily available.

So, I think we should stop rebooting the host after a boot failure, period, and wait for messages over the network. The boot failure is stored in a place the control plane can get to it (in the control-plane-agent). If we had a way of proactively sounding an alarm, we could do that, but for now it'd have to be polled.

Concretely, I discussed this briefly with @wesolows and the simplest thing appears to be:

  1. Honor reboot requests from the host normally, except
  2. If we get a host boot failure message, set a flag that causes the next reboot request to be interpreted as "power down and intervene."

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the host boot-failure handling and subsequent reboot-request path described in the issue. Identify where state can record the failure and where the next reboot request is interpreted; done means that request powers down for intervention rather than rebooting automatically, while ordinary reboot requests remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.