oxidecomputer / oxidecomputer/omicron

`KstatSampler` could report missing self-stat samples

Open
#8,025 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Metrics
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

The KstatSampler object is used to publish oximeter statistics based on illumos kstats. It includes a few specific bits of information about its own inner workings, such as the number of samples dropped for each tracked object. It places those on a queue, from which oximeter collects data periodically. That's a tokio::sync::broadcast::Sender, which acts like a limited ringbuffer to avoid egregious memory consumption if oximeter doesn't come knocking frequently enough.

In that case, where we've missed samples, the sampler currently logs a message:

https://github.com/oxidecomputer/omicron/blob/f5b92b7cf1efcc2b4f3f8352d847ad0ffaaa9f11/oximeter/instruments/src/kstat/sampler.rs#L1319-L1326

It would be nice to record this as its own timeseries, something like kstat_sampler:self_stat_samples_dropped. This needs to be a generic number of self-stat samples we've dropped, not attached to any particular tracked target. At the point we get this error, we don't have the actual dropped Sample, only the count, so it's a general-purpose thing about the KstatSampler itself, not any target it tracks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in oximeter/instruments/src/kstat/sampler.rs around lines 1319-1326, where dropped self-stat samples are currently logged. Trace how KstatSampler publishes its own statistics, then add a generic kstat_sampler:self_stat_samples_dropped timeseries using the dropped count rather than a tracked target. Done means the sampler records these missed self-stat samples as their own timeseries.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.