oxidecomputer / oxidecomputer/omicron
nexus did not come up with one switch missing
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
During today's dogfood update, all sleds apart from 14 were updated in parallel as the first step and then sled 14 was updated separately.
While that last mupdate was running, we expected the rack to come up and be accessible but found that nexus services were stuck attempting to talk to the missing switch:
07:20:33.806Z DEBG 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): client request
body = None
method = GET
uri = http://[fd00:1122:3344:10b::2]:12225/local/switch-id
07:20:48.808Z DEBG 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): client response
result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:10b::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, source: TimedOut })
07:20:48.808Z WARN 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): failed to identify switch slot for dendrite, will retry in 2 seconds
file = nexus/src/app/mod.rs:877
reason = Communication Error: error sending request for url (http://[fd00:1122:3344:10b::2]:12225/local/switch-id): operation timed out
zone_address = fd00:1122:3344:10b::2
Once sled 14 came up, nexus finished starting up without intervention but this experience indicates either that nexus cannot properly start if a scrimlet is unavailable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at nexus/src/app/mod.rs:877 and trace the startup path that retries the /local/switch-id request when a scrimlet or switch is unavailable. Reproduce the missing-switch sequence described in the issue and establish the expected startup behavior; done when that behavior is defined and the observed failure is addressed or covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100