oxidecomputer / oxidecomputer/omicron

sled-agent should reject instance requests if it receives them too early after startup

Open
#9,095 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

While testing updates on dublin, I had an instance running on a sled which was rebooted for an OS update. When the sled came back, Nexus reincarnated the instance back onto that same sled, but before the sled was actually ready to start instances. I'll attach the full sled-agent log; here's a timeline of some events present in it (mostly described in 1986 time, since things happened before time sync):

  • 1986-12-28 00:00:57 sled-agent starts
  • 1986-12-28 00:01:00.348Z adopt the M.2 disks where ledgers are stored
  • 1986-12-28 00:01:04.055Z SledAgent::new() starts
  • 1986-12-28 00:01:06.136Z config-reconciler reads ledgered OmicronSledConfig and starts its first reconciliation pass
  • 1986-12-28 00:01:06.151Z dropshot server starts listening on the underlay network
  • 1986-12-28 00:01:07.779Z respond with 404s to Nexus requests about the state of VMMs that were present before the reboot
  • 1986-12-28 00:01:09.225Z config-reconciler starts adopting U.2 disks
  • 1986-12-28 00:01:12.629Z return 200 for PUT /vmms/f325c556-9c3a-46bc-b716-84274dc99df5 with InitialInstanceState
  • 1986-12-28 00:01:12.760Z instance manager fails to set up the propolis zone because there are no available zone root datasets (logged as U2NotFound)
  • 1986-12-28 00:01:12.761Z VMM f325c556-9c3a-46bc-b716-84274dc99df5 is now in state failed with time_updated: 1986-12-28T00:01:12.760522953Z
  • 1986-12-28 00:01:37.224Z config-reconciler finishes adopting U.2 disks
  • 1986-12-28 00:01:37.691Z starts creating the first transient zone root dataset (parent of propolis zones)
  • (no timestamp available for when transient zone root datasets completed, but roughly judging from nearby logs it was a few seconds at most)
  • 1986-12-28 00:02:03.582Z start booting NTP zone
  • 1986-12-28 00:02:17.017Z last timestamp in 1986 before time syncs

I'm not sure when sled-agent should say it's definitely okay to start a zone; certainly it knows it can't succeed if there aren't any transient zone root datasets available yet. Maybe it should also wait for timesync? But we likely shouldn't have returned a 200 for the InitialInstanceState, and we should probably also have a clearer way to indicate "we're not ready yet" than that we happened to log U2NotFound in this case.

sled-agent.log

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing sled-agent startup through SledAgent::new(), config-reconciler disk adoption, transient zone root dataset creation, and the PUT /vmms/{id} handling described in the timeline. Determine where readiness should be represented and how an early instance request should be reported; done means requests are not accepted before the sled can start zones and the behavior is covered by the relevant sled-agent tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.