oxidecomputer / oxidecomputer/omicron
sled-agent should reject instance requests if it receives them too early after startup
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
While testing updates on dublin, I had an instance running on a sled which was rebooted for an OS update. When the sled came back, Nexus reincarnated the instance back onto that same sled, but before the sled was actually ready to start instances. I'll attach the full sled-agent log; here's a timeline of some events present in it (mostly described in 1986 time, since things happened before time sync):
1986-12-28 00:00:57sled-agent starts1986-12-28 00:01:00.348Zadopt the M.2 disks where ledgers are stored1986-12-28 00:01:04.055ZSledAgent::new()starts1986-12-28 00:01:06.136Zconfig-reconciler reads ledgeredOmicronSledConfigand starts its first reconciliation pass1986-12-28 00:01:06.151Zdropshot server starts listening on the underlay network1986-12-28 00:01:07.779Zrespond with 404s to Nexus requests about the state of VMMs that were present before the reboot1986-12-28 00:01:09.225Zconfig-reconciler starts adopting U.2 disks1986-12-28 00:01:12.629Zreturn 200 forPUT /vmms/f325c556-9c3a-46bc-b716-84274dc99df5withInitialInstanceState1986-12-28 00:01:12.760Zinstance manager fails to set up the propolis zone because there are no available zone root datasets (logged asU2NotFound)1986-12-28 00:01:12.761ZVMMf325c556-9c3a-46bc-b716-84274dc99df5is now in state failed withtime_updated: 1986-12-28T00:01:12.760522953Z1986-12-28 00:01:37.224Zconfig-reconciler finishes adopting U.2 disks1986-12-28 00:01:37.691Zstarts creating the first transient zone root dataset (parent of propolis zones)- (no timestamp available for when transient zone root datasets completed, but roughly judging from nearby logs it was a few seconds at most)
1986-12-28 00:02:03.582Zstart booting NTP zone1986-12-28 00:02:17.017Zlast timestamp in 1986 before time syncs
I'm not sure when sled-agent should say it's definitely okay to start a zone; certainly it knows it can't succeed if there aren't any transient zone root datasets available yet. Maybe it should also wait for timesync? But we likely shouldn't have returned a 200 for the InitialInstanceState, and we should probably also have a clearer way to indicate "we're not ready yet" than that we happened to log U2NotFound in this case.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing sled-agent startup through SledAgent::new(), config-reconciler disk adoption, transient zone root dataset creation, and the PUT /vmms/{id} handling described in the timeline. Determine where readiness should be represented and how an early instance request should be reported; done means requests are not accepted before the sled can start zones and the behavior is covered by the relevant sled-agent tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100