oxidecomputer / oxidecomputer/propolis
VMs with more than 254 vCPUs
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 270
- Forks
- 42
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 6
Description
Going up to 254 vCPUs is straightforward enough, and with a recent Helios change we're free to bump Propolis' MAXCPU up to 254, but going beyond that will involve some work.
- There's the (easier) control plane side, where the Propolis API currently wants a u8 of vCPUs. That won't do in the bold manymore core future.
- There's the (maybe easy, maybe hard) side, where we'll want guests to be able to direct interrupts to high-number CPUs if they're so inclined.
The obvious and somewhat daunting solution would be for Propolis to grow an IOMMU which can be used to let guests remap interrupts towards high-number cores like you'd do on real hardware.
https://david.woodhou.se/ExtDestId.pdf describes an alternative that might be workable: Hyper-V, KVM, FreeBSD byhve, and Xen all support a hypervisor feature to expand APIC IDs as used in MSI, but guest OS support is more mixed. Reportedly Linux supports this, Windows would be in a weird state either way, and the FreeBSD issue linked there is still open. I assume it would be mildly better for everyone to help get OSes in a good state with this than to build an IOMMU for Propolis, when the need comes..
Additionally, on Turin (and later? time will tell!) there's support for IOMMU virtualization. That is, the MMIO EFR register has vIommuSup set, as well as GTSup. that suggests maybe we can just .. offer a virtualized IOMMU and be done with it? given the IOMMU spec is 313 pages and I'm not too directly familiar with the AMD IOMMU I'd have questions for other knowers before being too sure, but this seems like a promising option for us. presumably this would mean Propolis' device emulation would have to be aware of any address translation the emulated devices ought to be doing? maybe there's something clever we can do about that.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the vCPU field in crates/propolis-api-types-versions/src/initial/components/board.rs and trace how the u8 limit flows through the control plane. Then investigate the interrupt-routing and virtualized-IOMMU options described in the issue, including guest support; done requires a decided, implementable path for supporting more than 254 vCPUs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100