oxidecomputer / oxidecomputer/omicron
Dropped metrics from ddmd and mgd
Open
@jmcarp is already working on this.
Since Jul 2, 2026.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Oximeter is currently dropping a subsample of metrics on the colo rack:
Investigating further, these failures all come from two producer ports, 8001 and 4677:
These metrics come from ddmd and mgd, and sure enough, there are gaps (note that these metrics should be sampled every 1s):
I think this means that collecting from these producers is often taking >1s. I think we should either figure out why and fix it, or figure out oximeter can better tolerate collection latencies greater than the sampling interval, or raise the sampling interval.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.