oxidecomputer / oxidecomputer/propolis

nvme: aggregate interrupts to not flood busy VMs

Open
#989 0 comments 0 reactions 1 assignee View on GitHub

@iximeow is already working on this.

Since Dec 17, 2025.

enhancement local storage storage
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

when we've pushed a CQE and completed an I/O, we currently always fire the interrupt for that completion queue. this comes with all the work of the guest handling the interrupt, exits to fiddle with LAPIC, returning to the VM, etc. we can cut this down somewhat by deferring a completion interrupt for 100-200 microseconds: if we're handling 600k I/Os/sec for a disk across 8 queues, that's 75k interrupts/sec/queue.

if we had batches of 200 microseconds on completions we could deliver those completions with only 5k interrupts/sec (assuming we were OK delivering 15 completions at once, which I am, at least). with a draft patch, I've gotten upwards of 2.2M I/Os/sec to a no-op backend, and there are still some edges I'd like to improve on.

this is independent of but related to Interrupt Coalescing as defined in the NVMe Base Specification (5.27.1.6 in 2.0a), which describes how we could expose coalescing settings to a VM, but we don't need to. From NVMe over PCIe Transport Specification, Interrupts says:

This value is provided to the controller as a recommendation by the host [...] The specific manner in which this value is used by the interrupt aggregation algorithm implemented by a controller is implementation specific.

so the fact that we return zeroes for the feature is not binding, and even if we accepted interrupt coalescing parameters, those are non-binding too.

where I'm fuzzy on is how a guest like Linux would set up interrupts across a 8 disks each with 32 queues, or something like that. at some point I'd expect the OS to have multiple queues on the same interrupt, and presumably we'd really want to scope coalescing across completion queues on that interrupt? but that is a bit more looking than I'm doing right now, probably a followup if anything.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.