oxidecomputer / oxidecomputer/omicron
Nexus should better-manage vCPU limits
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
sled-agent faithfully (and correctly!) reports all available processors as usable_hardware_threads. We then sum hardware threads for VMMs on that sled and vCPUs we'd like to place in the form of a new instance and check this sum is at or less than usable_hardware_threads.
... this means that for a Gimlet, as an example, we'll allow two 64-vCPU instances to be colocated on that Gimlet. This oversubscribes CPUs between VMs, Crucible, device emulation, kernel, and control plane services on the sled. Not ideal!
We don't currently limit VM-eligible threads below hardware threads at all, but we probably should. How that limiting should happen is more of an open question, compare with RFD 413 as an example of memory budgeting. CPU time-slicing makes the problem a bit more flexible at least. While an exact measure is difficult* at least an initial measure could inform a basic "set aside 16 CPUs"-style approach.
*: when does a service max out on CPU use? how much CPU might a CRDB use? Oximeter query load is transient! What about Nexus? how much do we earmark versus typical use? what about typical vs maximum Crucible load? or network? ... etc
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sled-agent/src/sled_agent.rs around usable_hardware_threads and nexus/db-queries/src/db/queries/sled_reservation.rs around the reservation check. Read RFD 413 for the memory-budgeting comparison, then investigate CPU use by the listed sled services and workloads. Done requires a decided CPU-budgeting approach and corresponding limits, rather than the current open-ended hardware-thread check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100