microsoft / microsoft/openvmm

nvme_driver: nvme device interrupts not spread across CPUs

Open
#1,664 1 comment 1 reaction 1 assignee View on GitHub

Nobody has claimed this yet.

storage
Dominant language
Rust
Stars
1.9k
Forks
238
Avg merge
1d 15h
Merged PRs (30d)
100

Description

NVMe devices have a fixed number of interrupt vectors (IVs). The nvme_driver creates one IoIssuer and IO Queue Pair per interrupt vector. When the number of IVs is less than the number of vCPUs, some vCPUs need to share the same IOQP and IV. The nvme_driver creates them in a greedy fashion, based on the CPU on which the IO was issued by the guest. In certain guest workloads. this means that each NVMe device in OpenHCL can overlap on a relatively small subset of CPUs. This problem becomes amplified when NVMe devices are used to support a striped disk: a single IO (say a write on CPU0), can cause multiple NVMe devices to create an IO issuer on CPU 0.

We should change the algorithm to not overload a subset of CPUs. Some options:

  1. Option 1: the vtl2 settings worker knows now many nvme devices an OpenHCL VM will have. When those settings are supplied, create a global cap. Respect that cap in the nvme_driver.
  2. Option 2: the nvme_driver can keep a tally of the number of IVs assigned to any given CPU. When the max vs. min count becomes too great, then find a "close" CPU.
  3. Option 3: The supported number of IVs and vCPUs is known. "spread out" the IVs across CPUs by generating a stride. Be careful not to always start this stride on the same CPU for all nvme devices.

In general, fallback to close CPUs should try to preserve NUMA locality.

Reported-By: @fliang-ms

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.