oxidecomputer / oxidecomputer/omicron

Possible race between HardwareManager and bootstore initialization

Open
#3,815 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When BootstrapAgent configures and starts the bootstore, it calls a couple of functions that themselves call StorageResources::all_m2_mountpoints(). The helpers will fail if no M.2 mount points are returned, but only require that at least one is present.

Immediately prior to configuring and starting the bootstore, BootstrapAgent waits for the storage manager to know about M.2 we booted from. However, we do not (and cannot in the general case, I think?) wait for the other M.2 to show up (since we don't even know if it's present / functional). If the other M.2 is physically present but storage manager doesn't find out about it until just after BootstrapAgent configures the bootstore, the bootstore will proceed using only the boot M.2; there's currently no way to come back later and add/reconcile the other M.2.

This isn't specific to the bootstore - it's just the first thing that tries to access the M.2s after BootstrapAgent has found the boot M.2. Reading from the Ledger (immediately after the bootstore is started) also uses all_m2_mountpoints, so is presumably subject to the same issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sled-agent/src/bootstrap/agent.rs at the storage-manager wait and bootstore configuration paths, then inspect sled-agent/src/storage_manager.rs and StorageResources::all_m2_mountpoints(). Trace how Ledger access follows bootstore startup. Done means the initialization path handles an M.2 device that becomes visible after the boot device without losing the opportunity to use it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.