oxidecomputer / oxidecomputer/omicron

nexus did not come up with one switch missing

Open
#4,724 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

During today's dogfood update, all sleds apart from 14 were updated in parallel as the first step and then sled 14 was updated separately.
While that last mupdate was running, we expected the rack to come up and be accessible but found that nexus services were stuck attempting to talk to the missing switch:

07:20:33.806Z DEBG 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): client request
    body = None
    method = GET
    uri = http://[fd00:1122:3344:10b::2]:12225/local/switch-id
07:20:48.808Z DEBG 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): client response
    result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:10b::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, source: TimedOut })
07:20:48.808Z WARN 20b100d0-84c3-4119-aa9b-0c632b0b6a3a (ServerContext): failed to identify switch slot for dendrite, will retry in 2 seconds
    file = nexus/src/app/mod.rs:877
    reason = Communication Error: error sending request for url (http://[fd00:1122:3344:10b::2]:12225/local/switch-id): operation timed out
    zone_address = fd00:1122:3344:10b::2

Once sled 14 came up, nexus finished starting up without intervention but this experience indicates either that nexus cannot properly start if a scrimlet is unavailable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at nexus/src/app/mod.rs:877 and trace the startup path that retries the /local/switch-id request when a scrimlet or switch is unavailable. Reproduce the missing-switch sequence described in the issue and establish the expected startup behavior; done when that behavior is defined and the observed failure is addressed or covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.