oxidecomputer / oxidecomputer/omicron
Multirack: On new racks Oximeter should not try to register with Nexus until told to do so
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
On single rack bringup, as soon as sled-agent starts, Oximeter tries to register with Nexus. This makes sense, as handoff has occured after RSS and Oximeter will be allowed to discover Nexus.
However on a new rack, when multirack join starts sled-agent, sled-agents are not supposed to be doing "active work". Sled-agents don't register themselves and instead wait for Nexus to discover them and only add them to the sled table in CRDB on an operator's command. Oximeter registration should wait for this point.
We can see that this is currently not the case when performing a multirack join command on a new rack in voxel with the code in #10989:
q16:59:57.206Z WARN SledAgent: failed to register with Nexus, will retry
count = 17
delay = 1486.727669497s
error = proto error: io error: No route to host (os error 148)
file = oximeter/producer/src/lib.rs:374
sled_id = 964383a1-fbe0-4b6a-9ba3-ce1a4880d5c1
We don't yet have the two racks connected via DDM over the front ports and so Oximeter consistently fails to register. This is fine, as it's just a nuisance in the logs. However, once we connect the racks, Oximeter will be able to find Nexus and register itself even though the sled-agents on the new rack aren't part of the multirack cluster yet.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at oximeter/producer/src/lib.rs around line 374 and trace how Oximeter registration is triggered during a multirack join. Compare that path with the operator-controlled point at which a new rack's sled-agents become part of the multirack cluster; done means Oximeter does not register before that point while single-rack bringup still behaves as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100