lablup / lablup/backend.ai

Eliminate the CUDA runtime API (libcudart) dependency from the cuda_open plugin using the driver API and NVML

Open
#13,598 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
21h 49m
Merged PRs (30d)
404

Description

## Motivation

The `cuda_open` plugin (`src/ai/backend/accelerator/cuda_open/nvidia.py`) loads `libcudart` (the CUDA **runtime** API), which is a CUDA _toolkit_ component. For the dockerized agent (BA-7264 / BA-7271) this is a problem:

- The toolkit is not present in the agent container unless bundled — and toolkit base images cost dozens of GB of storage pressure.
- A bundled `libcudart` must be ≤ the host driver's supported CUDA version, which cannot be known at image-build time → version-match fragility.

By contrast, `libcuda.so.1` (driver API) and `libnvidia-ml.so.1` (NVML) are **driver** components that `nvidia-container-runtime` injects into any container started with the NVIDIA runtime — always version-matched to the host, zero bytes bundled.

The plugin only uses the _device-management_ subset of the runtime API (never kernels/streams/memory ops), and every call has a 1:1 driver-API/NVML equivalent.

## Details

Replace `libcudart` in `nvidia.py` with a small `libcuda` binding (~8 functions); NVML usage stays as-is:

|cudart call|Consumed data|Replacement|
|---|---|---|
|`cudaGetDeviceCount`|device count|`cuDeviceGetCount`|
|`cudaGetDeviceProperties`|`name`|`cuDeviceGetName`|
| |`uuid`|`cuDeviceGetUuid_v2`|
| |`totalGlobalMem`|`cuDeviceTotalMem_v2`|
| |`multiProcessorCount`|`cuDeviceGetAttribute(CU_DEVICE_ATTRIBUTE_MULTIPROCESSOR_COUNT=16)`|
|`cudaDeviceGetPCIBusId`|PCI bus ID string|`cuDeviceGetPCIBusId`|
|`cudaRuntimeGetVersion`|`cuda_version` in agent info|`cuDriverGetVersion` (or NVML `nvmlSystemGetCudaDriverVersion_v2`)|
|`cudaDeviceReset`|**dead code** — no call sites|delete|

Notes:

1. `multiProcessorCount` is why libcuda (not NVML alone) is needed: NVML has no whole-GPU SM count (`nvmlDeviceGetNumGpuCores` returns CUDA cores, a different number).
1. **Keep enumeration on the driver API, not NVML**: both cudart and libcuda honor `CUDA_VISIBLE_DEVICES` and default to fastest-first ordering, while NVML enumerates in PCI order — NVML-only enumeration could silently renumber `DeviceId`s on heterogeneous multi-GPU hosts.
1. The ~450 lines of per-CUDA-version `cudaDeviceProp` ctypes structs (v10–v13) get deleted. They are a maintenance hazard: the v12 struct currently lists `accessPolicyMaxWindowSize` twice where the real struct has `maxBlocksPerMultiProcessor` — harmless today only because all consumed fields sit earlier in the layout. Driver attribute queries are version-stable ints.
1. Semantic change: the reported `cuda_version` becomes the driver-supported max CUDA version (what `nvidia-smi` shows) instead of the installed-toolkit version. In a containerized agent there is no host toolkit, so this is the correct value to report.
1. `cuInit` fails with an error code on GPU-less nodes — map onto the existing "library load failed → CUDA acceleration disabled" path.
1. Containerized agent deployment contract (document alongside BA-7272): `runtime: nvidia`, `NVIDIA_VISIBLE_DEVICES=all`, `NVIDIA_DRIVER_CAPABILITIES=compute,utility` (`utility` for NVML, `compute` for libcuda).

## Success Criteria

- [ ] `libcudart` is no longer loaded anywhere in the plugin; device discovery and stats work via libcuda + libnvml only.
- [ ] Device IDs, reported model names, memory sizes, SM counts, PCI IDs, and UUIDs are identical to the previous implementation on a bare-metal agent.
- [ ] The plugin works inside a container running with the NVIDIA container runtime without any CUDA toolkit installed.
- [ ] GPU-less hosts still degrade gracefully to "CUDA acceleration is disabled".

JIRA Issue: BA-7275

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/accelerator/cuda_open/nvidia.py and trace the existing libcudart loading, device enumeration, and statistics paths before reviewing the corresponding libcuda and NVML entry points. Done means libcudart and the unused reset path are gone, the listed device values remain equivalent, GPU-less hosts degrade gracefully, and the container runtime contract is documented alongside BA-7272.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, infrastructure
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.