oxidecomputer / oxidecomputer/omicron

Nexus could better manage in-memory inventory collections

Open
#5,296 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Good for new hires
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

There are two related problems (neither likely very urgent):

  • Inventory collections are currently irregularly spaced in time. There are three Nexus instances, each of which carries out a collection every 10 minutes. So we tend to get three collections about every 10 minutes.
  • Currently, various processes that need an inventory collection fetch the latest one from the database. This is generally fine, but it's not a super cheap operation and likely to get more expensive as inventories and systems grow.

Here's a proposal that improves both problems:

  • Create a background task "inventory_fetch" that just fetches the latest collection and makes it available to any other part of Nexus that wants it (via a watch channel). This could run fairly often, particularly if it doesn't bother re-fetching a collection if it's still the latest one. e.g., every 15 seconds, Nexus could check if it has the latest collection, and only if it doesn't would it re-fetch it. This task should explicitly depend on the current "inventory collection" background task that actually fetches the inventory so that whenever Nexus successfully collects it, we immediately activate this task. (I would still have inventory_fetch fetch the collection that was just created fro the database to ensure that it uses the same code path whether or not this Nexus was the one that collected that one.)
  • Change the existing inventory_collector background task to (1) run more frequently (e.g., every 30 seconds) and (2) do nothing if it was awakened by a timeout and the most recent collection (from inventory_fetch) is "new enough" (e.g., from within the last two minutes, maybe plus/minus some random interval to attempt to stagger collections from multiple Nexus instances).

I think this would be pretty straightforward and would have a few advantages:

  • We don't need to re-fetch inventory collections for every operation that wants to use them. Nexus at-large can assume it always has a pretty recent inventory collection in-memory.
  • We can control fairly precisely the latency from any Nexus successfully collecting inventory to all Nexus instances knowing about it (assuming no failures) by tuning the period of the inventory_fetch task.
  • We can control fairly precisely how old the latest collection could be (again assuming no failures) by directly tuning the "new enough" threshold. Admittedly, you can sort of do this today by tuning the period of the collector task, but only if we're willing to do up to 3x the work (one for each Nexus instance).
  • We'd reduce duplicate work from multiple Nexus instances when nothing's actually changed but they all happen to decide to collect around the same time.

I don't think any of these problems is that urgent though.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the inventory_collector background task and the existing database fetch path for inventory collections. Trace how background tasks are scheduled and how consumers request the latest collection. Done means the proposed inventory_fetch flow, in-memory sharing, collector scheduling, freshness checks, and tests are implemented without unnecessary duplicate collections.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.