oxidecomputer / oxidecomputer/omicron

The background task subsystem could expose basic telemetry information

Open
#4,870 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Metrics nexus
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I was talking with @davepacheco about our existing metrics, and the idea of publishing some data from the background task subsystem came up. It seems like both valuable information and a reasonably well-understood component. The background task Driver starts a new tokio task for each registered background task:

https://github.com/oxidecomputer/omicron/blob/030adce411fe37c9e2d3c70ee5a6cdbdfd49f3f9/nexus/src/app/background/common.rs#L259-L261

That TaskExec is mostly a wrapper around a tokio select loop:

https://github.com/oxidecomputer/omicron/blob/030adce411fe37c9e2d3c70ee5a6cdbdfd49f3f9/nexus/src/app/background/common.rs#L369-L385

That waits for one of three events:

  • An explicit notification
  • A timeout
  • All of its dependencies having finished

We already store a lot of information about the completion status of these tasks, here:

https://github.com/oxidecomputer/omicron/blob/030adce411fe37c9e2d3c70ee5a6cdbdfd49f3f9/nexus/src/app/background/common.rs#L430

This set of data is a great candidate for publishing to oximeter. We could imagine sending the duration of each task's execution, along with:

  • The task name
  • The reason it ran (the ActivationReason enum, basically)
  • The outcome (success or failure, might be enough, though we do have the "details" as a JSON blob)

Making this easily available through a dashboard and / or omdb would be sweet too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/src/app/background/common.rs, examining Driver, TaskExec, the tokio select loop, and the completion status data around the referenced lines. Determine how execution duration, task name, ActivationReason, and outcome could be published to oximeter, then clarify whether dashboard or omdb exposure is part of done.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.