oxidecomputer / oxidecomputer/omicron

Multirack: On new racks Oximeter should not try to register with Nexus until told to do so

Open
#11,094 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

multi-rack
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

On single rack bringup, as soon as sled-agent starts, Oximeter tries to register with Nexus. This makes sense, as handoff has occured after RSS and Oximeter will be allowed to discover Nexus.

However on a new rack, when multirack join starts sled-agent, sled-agents are not supposed to be doing "active work". Sled-agents don't register themselves and instead wait for Nexus to discover them and only add them to the sled table in CRDB on an operator's command. Oximeter registration should wait for this point.

We can see that this is currently not the case when performing a multirack join command on a new rack in voxel with the code in #10989:

q16:59:57.206Z WARN SledAgent: failed to register with Nexus, will retry
    count = 17
    delay = 1486.727669497s
    error = proto error: io error: No route to host (os error 148)
    file = oximeter/producer/src/lib.rs:374
    sled_id = 964383a1-fbe0-4b6a-9ba3-ce1a4880d5c1

We don't yet have the two racks connected via DDM over the front ports and so Oximeter consistently fails to register. This is fine, as it's just a nuisance in the logs. However, once we connect the racks, Oximeter will be able to find Nexus and register itself even though the sled-agents on the new rack aren't part of the multirack cluster yet.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at oximeter/producer/src/lib.rs around line 374 and trace how Oximeter registration is triggered during a multirack join. Compare that path with the operator-controlled point at which a new rack's sled-agents become part of the multirack cluster; done means Oximeter does not register before that point while single-rack bringup still behaves as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.