oxidecomputer / oxidecomputer/omicron
Provide pre-nexus observability API
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Things can go sideways during RSS, upgrades, and cold boots before Nexus is up. We need a way for operators to observe what's going on and take corrective action in the absence of the primary API being available.
A concrete example of this is an NTP misconfiguration. This happened to me the other day. I set ntp_servers in the RSS config to an NTP server that only exists on our VPN, but I was running from home on a machine without access to the VPN. I wound up running chronyc tracking in the NTP zones to ultimately figure out what was going wrong. But this situation can very easily happen in a real deployment environment for a number of reasons. And we need a way for operators to debug these kinds of issues.
One possible way to do this would be to provide an observability API in wicketd that's accessible via the technician ports.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining wicketd and how its technician ports are exposed, then identify what operators need to observe and control before Nexus is available. The work is done when a pre-Nexus observability API supports diagnosing cases such as NTP misconfiguration and provides corrective actions during RSS, upgrades, and cold boots.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100