oxidecomputer / oxidecomputer/omicron
Nexus could better manage in-memory inventory collections
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
There are two related problems (neither likely very urgent):
- Inventory collections are currently irregularly spaced in time. There are three Nexus instances, each of which carries out a collection every 10 minutes. So we tend to get three collections about every 10 minutes.
- Currently, various processes that need an inventory collection fetch the latest one from the database. This is generally fine, but it's not a super cheap operation and likely to get more expensive as inventories and systems grow.
Here's a proposal that improves both problems:
- Create a background task "inventory_fetch" that just fetches the latest collection and makes it available to any other part of Nexus that wants it (via a
watchchannel). This could run fairly often, particularly if it doesn't bother re-fetching a collection if it's still the latest one. e.g., every 15 seconds, Nexus could check if it has the latest collection, and only if it doesn't would it re-fetch it. This task should explicitly depend on the current "inventory collection" background task that actually fetches the inventory so that whenever Nexus successfully collects it, we immediately activate this task. (I would still haveinventory_fetchfetch the collection that was just created fro the database to ensure that it uses the same code path whether or not this Nexus was the one that collected that one.) - Change the existing
inventory_collectorbackground task to (1) run more frequently (e.g., every 30 seconds) and (2) do nothing if it was awakened by a timeout and the most recent collection (frominventory_fetch) is "new enough" (e.g., from within the last two minutes, maybe plus/minus some random interval to attempt to stagger collections from multiple Nexus instances).
I think this would be pretty straightforward and would have a few advantages:
- We don't need to re-fetch inventory collections for every operation that wants to use them. Nexus at-large can assume it always has a pretty recent inventory collection in-memory.
- We can control fairly precisely the latency from any Nexus successfully collecting inventory to all Nexus instances knowing about it (assuming no failures) by tuning the period of the
inventory_fetchtask. - We can control fairly precisely how old the latest collection could be (again assuming no failures) by directly tuning the "new enough" threshold. Admittedly, you can sort of do this today by tuning the period of the collector task, but only if we're willing to do up to 3x the work (one for each Nexus instance).
- We'd reduce duplicate work from multiple Nexus instances when nothing's actually changed but they all happen to decide to collect around the same time.
I don't think any of these problems is that urgent though.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the inventory_collector background task and the existing database fetch path for inventory collections. Trace how background tasks are scheduled and how consumers request the latest collection. Done means the proposed inventory_fetch flow, in-memory sharing, collector scheduling, freshness checks, and tests are implemented without unnecessary duplicate collections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100