oxidecomputer / oxidecomputer/omicron

Sled-level resource metrics

Open
#9,559 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

As an operator, I want to be able to understand the state of various physical resources on the rack. Is a physical cpu core heavily utilized, or waiting on another resource, or queueing operations? Is a disk almost full, or saturating IOPS? We have coverage for some of these metrics in oxql for virtual machines (virtual_machine:, virtual_disk:), but less for physical resources on the sled.

As @rmustacc pointed out on a call yesterday, there's a lot of nuance to consider here. For example, RFD 526 goes into great detail on just a single resource type. But there's also low-hanging fruit that can produce value to operators more quickly: we can add basic metrics before enumerating all the telemetry we eventually want to include.

One potential starting point is to identify physical resources of interest, and collect USE metrics (utilization, saturation, errors) (or analogously the four "golden signals") for each one. We could start with cpu, memory, disk, and network (although note that we already have some sled-level network metrics in sled_data_link:*).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing oxql virtual_machine:* and virtual_disk:* metrics, along with the sled_data_link:* metrics mentioned in the issue. Use RFD 526 and the USE method as context, then identify a scoped initial set of sled resources and the metrics needed to cover them; done means that scope and its implementation path are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.