[Feature]: Add a configurable `--sysfs-root` for device NUMA detection (`GetNumaNode`)

Open
#1,833 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
74/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Read internal/rm/nvml_devices.go:150-162 and internal/rm/devices.go:93-101 to trace the NUMA lookup and topology construction. Then locate the existing flag configuration for the --nvidia-*-root options and follow how a new --sysfs-root value would reach GetNumaNode. Done means the default remains /sys and a redirected synthetic sysfs tree supplies NUMA topology for simulated devices.

Written by the indexing model from the issue text.

Description

Summary

GetNumaNode reads a device's NUMA node from a hardcoded /sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.
This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same --sysfs-root request for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).

Problem

internal/rm/nvml_devices.go:150-162:

busID := strings.ToLower(strings.TrimPrefix(uint8Slice(info.BusId[:]).String(), "0000"))
b, err := os.ReadFile(fmt.Sprintf("/sys/bus/pci/devices/%s/numa_node", busID))
if err != nil {
    return false, 0, nil  // error swallowed -> device reported with no NUMA
}

The /sys prefix is a literal; nothing overrides it (--nvidia-driver-root / --nvidia-dev-root feed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails, hasNuma=false, and BuildDevice (internal/rm/devices.go:93-101) leaves Device.Topology nil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.

Proposed solution

Flag --sysfs-root (default /sys), matching #264 and the existing --nvidia-*-root family:

os.ReadFile(filepath.Join(sysfsRoot, "bus/pci/devices", busID, "numa_node"))

In-tree precedent for the same read: go-nvlib's WithPCIDevicesRoot (pkg/nvpci/nvpci.go:187,317, mock at mock.go:104) and k8s dynamic-resource-allocation's WithFSFromRoot (which #264 builds on). Default unchanged means zero impact on real deployments.

Alternatives

  • nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returns NVML_ERROR_NOT_SUPPORTED for it, and on real hardware it signals "no NUMA" via return code (not the sysfs -1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway.
  • Subpath-mount the tree over /sys/bus/pci/devices: possible (#264's fallback) but shadows the real PCI dir per-pod; a flag is cleaner.

Scoped to GetNumaNode (NVML full GPU + MIG, which delegates to the parent). A second hardcoded root at internal/vgpu/pciutil.go:42 (GFD vGPU detection) could adopt the same flag later. Happy to send a PR.

Dominant language
Go
Stars
3.9k
Forks
868
Avg merge
2d 14h
Merged PRs (30d)
27

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/k8s-device-plugin

All issues in NVIDIA/k8s-device-plugin

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.