oxidecomputer / oxidecomputer/omicron

Resource utilization metrics could report samples even when there are no changes

Open
#5,289 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Metrics
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Nexus currently keeps track of the number of resources provisioned, CPUs, memory, and disks. Those go into the utilization views in the web console:

Screen Shot 2024-03-19 at 11 17 22

As you can see there, the data is sampled irregularly. In particular, Nexus generates samples only when there are changes to the data. That is reasonable from an efficiency perspective, since it only reports deltas. However, it makes querying and the graph shown above more painful. It's impossible to know, for example, if any particular time range contains any data. That leads the console implementation to do grubby things like find the latest sample before and after the requested time range.

An alternative would be to report data for intervals even in which there are no changes. This greatly simplifies querying, graphing, and understanding the data, at the obvious expense of transferring more data. The size here is non-trivial, being linear in the number of distinct "virtual collections", which I believe means non-deleted projects, silos, and fleets.

One subtlety here is that we may still wish to report each change, not the total change in the sample period. For example, if a user provisions two new VMs in a sample period, do we report two samples, or one with the sum of the new provisioned resources? I'd expect we want the former, to avoid missing individual changes. So the reported samples would really be:

  • The last value, if there have been no changes
  • The value at each change, as a separate sample

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read nexus/db-model/src/virtual_provisioning_collection.rs and the linked console SystemMetric.tsx range logic first. Trace the Nexus sampling entry point, then settle how unchanged intervals emit the last value while preserving each change as its own sample; done means this behavior is implemented and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, typescript
Domain
backend, data-visualization, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.