oxidecomputer / oxidecomputer/omicron

Nexus should better-manage vCPU limits

Open
#8,632 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

sled-agent faithfully (and correctly!) reports all available processors as usable_hardware_threads. We then sum hardware threads for VMMs on that sled and vCPUs we'd like to place in the form of a new instance and check this sum is at or less than usable_hardware_threads.

... this means that for a Gimlet, as an example, we'll allow two 64-vCPU instances to be colocated on that Gimlet. This oversubscribes CPUs between VMs, Crucible, device emulation, kernel, and control plane services on the sled. Not ideal!

We don't currently limit VM-eligible threads below hardware threads at all, but we probably should. How that limiting should happen is more of an open question, compare with RFD 413 as an example of memory budgeting. CPU time-slicing makes the problem a bit more flexible at least. While an exact measure is difficult* at least an initial measure could inform a basic "set aside 16 CPUs"-style approach.

*: when does a service max out on CPU use? how much CPU might a CRDB use? Oximeter query load is transient! What about Nexus? how much do we earmark versus typical use? what about typical vs maximum Crucible load? or network? ... etc

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent/src/sled_agent.rs around usable_hardware_threads and nexus/db-queries/src/db/queries/sled_reservation.rs around the reservation check. Read RFD 413 for the memory-budgeting comparison, then investigate CPU use by the listed sled services and workloads. Done requires a decided CPU-budgeting approach and corresponding limits, rather than the current open-ended hardware-thread check.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.