GPU-accelerated Sandbox Support

Problem Statement

Some agent workloads need access to NVIDIA GPU hardware and drivers from inside an OpenShell sandbox. Examples include developing CUDA kernels, testing GPU software, or running model fine-tuning workloads.

OpenShell exposes GPU access through sandbox creation options, but the behavior needs to be made consistent across GPU-enabled compute drivers and covered by validation tests.

Scope

This roadmap is focused on NVIDIA GPU support for OpenShell sandboxes.

In-scope drivers:

• Docker
• Podman
• Kubernetes
• VM

Out of scope:

• Generic non-GPU device request flags. Track those with #628 and the resource requirements work in NVIDIA/OpenShell#1360.
• HAMi-specific or vGPU-provider-specific implementation. NVIDIA/OpenShell#1065 is a related use case that should be enabled by the richer resource model in NVIDIA/OpenShell#1360, not implemented directly here.
• Reintroducing ad hoc driver capacity fields. PR NVIDIA/OpenShell#1402 removes unused driver capability/capacity fields; richer capability and capacity APIs should be deferred to NVIDIA/OpenShell#1360.
• Historical CDI migration work from #398, which was filed for the older k3s-based GPU architecture and is now superseded by driver-specific GPU paths.

Target GPU Request Semantics

--gpu requests the active driver's default GPU behavior.

For GPU-enabled drivers, the target default behavior is to inject or allocate one suitable GPU, not all GPUs. The mechanism for selecting that GPU is driver-specific.

Target defaults:

• Docker: select one suitable CDI GPU device by default.
• Podman: select one suitable CDI GPU device by default.
• Kubernetes: request one nvidia.com/gpu resource and delegate placement to the scheduler/device plugin.
• VM: assign one available GPU from the VM driver inventory.

A "free GPU" is driver-specific. From OpenShell's perspective, a GPU is free if it is available according to the active driver's allocation model and is not already assigned by the same OpenShell gateway/driver in a conflicting way. OpenShell should not attempt to infer global host GPU usage by external processes or workloads.

For Kubernetes, free capacity is delegated to the scheduler and device plugin. Depending on device plugin configuration, such as time slicing or sharing, a schedulable GPU request may not mean an unused physical GPU.

Advanced Device Selection

--gpu-device remains driver-native for now.

This is an advanced option for users who understand the active driver's device identifiers:

• Docker/Podman: CDI device IDs such as nvidia.com/gpu=0
• VM: driver-supported GPU index or PCI BDF
• Kubernetes: scheduler/resource-driven behavior; explicit physical device selection is not a portable default

--gpu-device should not be the normal path for requesting GPU access. It exists as an escape hatch and is currently more important for Docker/Podman because their default --gpu behavior still maps to nvidia.com/gpu=all. Once Docker and Podman default to selecting one GPU, most users should prefer --gpu.

GPU Count

--gpu-count is a cross-driver roadmap goal.

Kubernetes support is tracked by NVIDIA/OpenShell#1338 and is being implemented in NVIDIA/OpenShell#1156. Docker, Podman, and VM count support should be tracked as separate implementation work.

Until a driver explicitly supports count-based allocation:

• gpu_count == 0 means unspecified.
• gpu_count > 0 must be rejected by that driver.
• Unsupported drivers must not silently ignore non-zero gpu_count.

Once Docker and Podman default --gpu behavior selects one GPU instead of nvidia.com/gpu=all, those drivers may treat --gpu-count=1 as equivalent to --gpu.

Support for --gpu-count=N where N > 1 still requires explicit driver implementation.

Validation

GPU support should be documented through e2e validation.

Validation categories include:

• device discovery, selection, and visibility, currently covered by nvidia-smi-based tests,
• execution validation, starting with a basic CUDA workload,
• future API-specific validation, such as OpenCL or Vulkan.

Docker is the first validation target because it is the most mature GPU-enabled e2e path. Podman should follow after Docker.

Tracked validation work:

• NVIDIA/OpenShell#1472: define GPU validation tests for GPU-enabled drivers.
• NVIDIA/OpenShell#1476: define GPU validation image artifacts.

Current Subtasks

• Assess Kubernetes GPU support preflight validation: NVIDIA/OpenShell#1807
• Define and run GPU validation tests for Docker, then expand to other GPU-enabled drivers: NVIDIA/OpenShell#1472
• Define driver-agnostic GPU validation image artifacts: NVIDIA/OpenShell#1476
• Enable CUDA GPU execution validation in GPU CI once a published CUDA validation image exists: NVIDIA/OpenShell#1487
• Fix default GPU filesystem policy so CUDA workloads can initialize under OpenShell sandbox policy: NVIDIA/OpenShell#1486
• Add Kubernetes --gpu-count support: NVIDIA/OpenShell#1338 / NVIDIA/OpenShell#1156
• Add Docker/Podman default CDI GPU selection so --gpu prefers one GPU instead of all GPUs: NVIDIA/OpenShell#1477
• Review and land VM GPU container support: NVIDIA/OpenShell#1143
• Derive sandbox access requirements from selected CDI specs: NVIDIA/OpenShell#1606
• Narrow GPU procfs permissions and surface runtime-added permissions: NVIDIA/OpenShell#1628

Related Work

• NVIDIA/OpenShell#1065: HAMi/vGPU sharing use case. Related, but not an explicit implementation target here.
• NVIDIA/OpenShell#1360: richer sandbox resource requirements model. Future resource/capacity and generic device work should build on this.
• NVIDIA/OpenShell#1402: trims unused compute driver capability response fields. Do not reintroduce ad hoc GPU capacity fields here.
• #628: generic sandbox device request flags. Related to NVIDIA/OpenShell#1360 and should be tracked under a separate generic resource/device roadmap.
• #398: historical CDI migration context for the older k3s-based GPU architecture. Closed as superseded by the current driver-specific GPU paths and remaining CDI work tracked in NVIDIA/OpenShell#1606.