nestybox / nestybox/sysbox-mgr

MIG GPU passthrough via DRA/CDI (deviceMgr + MIG cap-node mknod)

Open
#83 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
7
Forks
23
Avg merge
1h 49m
Merged PRs (30d)
1

Description

Need
System containers need access to NVIDIA MIG GPUs allocated through Kubernetes DRA/CDI. When the NVIDIA driver is containerized (GPU-Operator), the per-slice MIG capability nodes under /dev/nvidia-caps never materialize on the host — they only exist inside the driver container at /run/nvidia/driver/dev. sysbox runs containers in a user namespace and bind-mounts device nodes from a host path, so without these nodes the container setup fails and the workload crashes.

Proposal

  • Add a device manager (deviceMgr) that reconciles device discovery: a radix-tree of devicers (root/net/amd/intel/nvidia), per-container device-node creation under the sysbox lib dir, and DeviceMounts for bind-mounting into the container. The nvidia devicer reconciles container device paths against host roots {"/", "/run/nvidia/driver"} covering both direct /dev nodes and MIG caps under /dev/nvidia-caps.
  • Add isMigCapNode() to detect char nodes under /dev/nvidia-caps/nvidia-cap* and mknodMigCapNode() to create a missing node on the host from its (major, minor) pair — idempotent, invoked from Discover() when capability reads fail for a MIG cap node. sysbox-mgr runs as root, so host mknod is allowed.
  • Register the SetupDevices gRPC callback, remove devices on container teardown, and append device bind-mounts during mount request handling. Add go-immutable-radix dependency.

Benefits
Lets sysbox secure containers pass through NVIDIA MIG GPUs allocated via DRA/CDI, unblocking GPU-Operator-based MIG workspaces.

Part of a 3-repo feature (see sysbox-runc and sysbox-ipc PRs).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Discover(), the SetupDevices gRPC callback, container teardown, and mount request handling in sysbox-mgr. Verify how the nvidia devicer handles host roots and MIG capability nodes, then confirm that device discovery, host node creation, cleanup, and bind-mounts support DRA/CDI workloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.