oxidecomputer / oxidecomputer/hubris

`packrat` should latch the boot slot used for the current host boot

Open
#2,597 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

While working on #2518, we realized that it was difficult to report the boot slot used for a host that has panicked.

This is due to the fact that before panicking, the host could have switched the "next boot" slot to the other one, and we currently (AFAIK) can only obtain the slot selected for next boot.

While implementing #2518, I punted on this, always reporting None/"unknown slot" while obtaining the host panic message. In chat, we decided:

  • We do want this data, but the way it's currently being obtained is not necessarily accurate
  • For bootfail, the "currently selected" slot is correct, since a booting-host can't/won't swap the slots. For host panic, the "currently selected" slot may not actually match what the currently-panicking-host actually booted with, if it has changed the slot after successfully booting.
  • I will include basically Option<Slot> (or some equivalent) in both ereport and MGS message schemas
  • for now, "host panic" ereports/MGS messages will always have "None". I'll make an issue to figure out how to accurately "latch" the last booted slot somewhere, and store it off in packrat. We can fix this in a follow-up PR.
  • "bootfail" ereports will include a Some(Slot), unless drv-hf is panicked and I can't ask it which slot is selected (which would be pretty bad, but I still will write code to handle this), with the currently selected code.

This is the tracking issue to resolve this. In the current branch, the offending lines are here (this will change after 2518 is merged):

https://github.com/oxidecomputer/hubris/blob/2a771367a6b775e97c70a6304c34ff923cdfcddd/task/host-sp-comms/src/main.rs#L1009-L1011

We may also want to consider tracking some additional info, or invalidating the storage, if there have been subsequent host boots, e.g. if we do something like:

  • host boots from slot 0
  • host panics, 0 is stored
  • host boots
  • host writes slot 1
  • host boots
  • host writes slot 0
  • host boots
  • NOW host panic is obtained

We should maybe make it clear "the boot slot 0 referred to with the panic data isn't the same as the one reported in the panic", to avoid debugging confusion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in task/host-sp-comms/src/main.rs around lines 1009-1011, then trace how packrat stores boot-slot information and how host panic data is assembled. Determine where the last booted slot can be latched and whether later boots should invalidate it. Done means host panic ereports and MGS messages report the accurately latched slot or an explicit unknown value when it cannot be trusted.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.