NVIDIA / NVIDIA/NemoClaw

[Jetson Thor][Onboard] Documented GPU-device and CDI paths cannot pin a sandbox to MIG GI 1

Open
#9,154 1 comment 0 reactions 0 assignees View on GitHub
area: onboarding area: sandbox platform: jetson
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

### Investigation Summary

- On Jetson AGX Thor with MIG enabled, `nvidia-ctk cdi list` reports only `nvidia.com/gpu=0`, `nvidia.com/gpu=all`, `nvidia.com/pva=0`, and `nvidia.com/pva=all`. It does not list MIG UUIDs or `/dev/nvidia-caps` nodes.
- CSV-mode `nvidia-ctk cdi generate --device-id=` fails with `unsupported device id`. `--mode=nvml` selects `nvidia-cap12`/`nvidia-cap13` for GI 1, then fails because it cannot locate `libcuda.so.595.78`.
- `docker run --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=` fails with the same CSV `unsupported device id`. `--device nvidia.com/gpu=` is unresolvable. `--gpus` is rejected on this Jetson host (the runtime requires `--runtime=nvidia`).
- Installed NemoClaw v0.0.103 Jetson onboard uses the NVIDIA runtime and `NVIDIA_VISIBLE_DEVICES=`, so it hits that CSV path. OpenShell 0.0.85 `sandbox create` accepts `--gpu [COUNT]` only. It has no `--gpu-device`.
- A working pin is an unscoped OpenShell create, then a clone of that container with raw `--device` nodes for GI 1 (`nvidia-cap12` + `nvidia-cap13`). `docker compose up`/`down` on that sandbox unregisters it (`Policy fetch failed ... sandbox not found`). That crash is not the MIG device failure.

### Description

NemoClaw and OpenShell cannot pin a sandbox to Jetson Thor MIG GPU instance 1 (`2g.0gb+gfx`) through the documented GPU-device, CDI, or `NVIDIA_VISIBLE_DEVICES` path.

**Expected:** An operator can pin the sandbox to GI 1 and leave GI 2 (`1g.0gb+me`) unused. Documented selectors (`--sandbox-gpu-device`, CDI `nvidia.com/gpu=`, or `NVIDIA_VISIBLE_DEVICES=`) should either attach that partition or fail with a Jetson-specific error that names a supported pin method.

**Actual:** Every documented selector fails in NVIDIA Container Toolkit CSV mode. The Jetson compatibility path in NemoClaw v0.0.103 still sets `NVIDIA_VISIBLE_DEVICES` and therefore fails the same way. Public docs still tell operators to pass `--sandbox-gpu --sandbox-gpu-device ` into `openshell sandbox create`. OpenShell 0.0.85 does not implement `--gpu-device`.

**Working operator pin (do not treat this as a supported product interface):**

1. Create the sandbox with GPU off or unscoped (`openshell sandbox create` without a MIG UUID). Keep the gateway registration.
2. Clone that container in place with raw `--device` nodes for GI 1: `/dev/nvidia-caps/nvidia-cap12` (GI) and `/dev/nvidia-caps/nvidia-cap13` (CI 0), plus `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, and `/dev/nvmap` as needed.
3. Do not set `NVIDIA_VISIBLE_DEVICES`. Do not inject CDI. Isolation is the device cgroup.

**Do not recommend these on this JetPack / toolkit combination:**

- `--sandbox-gpu-device` with a MIG UUID or CDI name.
- `NVIDIA_VISIBLE_DEVICES=`.
- `nvidia-ctk cdi generate` without `--output=/tmp/...`. That rewrite of host CDI still emits only `0`/`all`.
- `libcuda.so.595.78` or `libnvidia-ml.so.595.78` symlinks. Toolkit then reports `unexpected version 595.78 != 1.1`.

**Related lifecycle failure (same investigation, different cause):** `docker compose up` or `docker compose down` against an OpenShell-managed sandbox unregisters it with the gateway. The sandbox then crash-loops with `Policy fetch failed ... sandbox not found` even when the MIG device nodes are correct. Orphaned credentials in a compose file reproduce that crash. Prefer `docker restart` or an in-place clone. Do not recreate the sandbox through Compose.

**Proposed product fix:**

- After an unscoped OpenShell create, pin GI 1 with a raw-devices clone (for example `--sandbox-gpu-mig-profile 2g.0gb+gfx`).
- Do not send MIG UUIDs through `--sandbox-gpu-device`, CDI, or `NVIDIA_VISIBLE_DEVICES`.
- Persist the pin without a Compose recreate so the gateway registration stays valid.
- v0.0.103 does not ship `--sandbox-gpu-mig-profile`. A later unreleased source change adds that flag. This issue tracks the released onboard gap.

**Related issues (not duplicates):**

- NVIDIA/NemoClaw#7610: IGX Orin non-root `cuInit` proof. Different platform and failure.
- NVIDIA/NemoClaw#8500: Jetson AGX Thor Express installer epic. GPU access on Thor is in that scope.
- NVIDIA/NemoClaw#3710: k3s nvidia-device-plugin CDI on older Jetson onboard. This host uses the Docker compatibility path.
- NVIDIA/OpenShell#1444: GPU sandbox / `--gpu-device` design. OpenShell 0.0.85 still has `--gpu [COUNT]` only.
- NVIDIA Container Toolkit CSV generation accepts device ids `all` and `0` only. Mixed CSV mode states that MIG ids are not supported. NVML generate on this L4T tree fails on `libcuda.so.1.1` versus `libcuda.so.595.78`.

### Reproduction Steps

1. On Jetson AGX Thor with JetPack L4T R39.2, enable MIG and create the two recommended GPU instances (profile 83 `2g.0gb+gfx` as GI 1, profile 78 `1g.0gb+me` as GI 2), then create default compute instances.
2. Confirm host inventory:

```bash
nvidia-smi -L
nvidia-ctk --version
nvidia-ctk cdi list
ls /dev/nvidia-caps
```

3. Try the documented CDI generate path with output under `/tmp` only:

```bash
sudo nvidia-ctk cdi generate --device-id=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 --output=/tmp/nvidia-cdi-mig.yaml
sudo nvidia-ctk cdi generate --device-id=0:1 --output=/tmp/nvidia-cdi-0-1.yaml
sudo nvidia-ctk cdi generate --mode=nvml --output=/tmp/nvidia-cdi-nvml.yaml
```

4. Try Docker injection:

```bash
docker run --rm --gpus device=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
docker run --rm --device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
```

5. Try the documented NemoClaw selector (v0.0.103):

```bash
nemoclaw onboard --name my-assistant --sandbox-gpu --sandbox-gpu-device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1
```

6. Compare OpenShell create flags:

```bash
openshell sandbox create --help
```

7. Optional lifecycle check: after a gateway-registered unscoped sandbox exists, run `docker compose up` or `docker compose down` against that container. The gateway then reports the sandbox as missing.

### Environment

- OS: Ubuntu 24.04.4 LTS (Noble), aarch64
- Hardware: NVIDIA Jetson AGX Thor, GPU 0 UUID `GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600`
- JetPack / L4T: R39.2.0 (GCID 45755727), kernel 6.8.12-1021-tegra
- Driver userspace: host `libcuda.so.1.1` under `/opt/nvidia/l4t-gpu-libs/openrm/` (no `libcuda.so.595.78` file)
- MIG GI 1: profile 83 `2g.0gb+gfx`, placement 0:2, UUID `MIG-314bde5a-0c51-524c-b404-7fe9c0231de1`, caps `/dev/nvidia-caps/nvidia-cap12` (GI) + `nvidia-cap13` (CI 0)
- MIG GI 2: profile 78 `1g.0gb+me`, placement 2:1, UUID `MIG-f177deec-7f6d-5cc3-88d9-e746daca1917`, caps `nvidia-cap21` + `nvidia-cap22` (left unused)
- Node.js: v22.23.2
- Docker: Docker Engine 29.7.2
- NVIDIA Container Toolkit CLI: 1.19.1, commit `598a9cbb6239c7d05df9c881bad36df7eff34d78`
- OpenShell: 0.0.85
- NemoClaw: v0.0.103 (`nemoclaw --version`)
- CDI: `/var/run/cdi/nvidia.yaml` devices `0` and `all` only; `/etc/cdi` contains `nvidia-pva.yaml` only
- CSV `devices.csv`: includes `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvmap`; does not include `/dev/nvidia-caps/*`

### Debug Output

The `nemoclaw debug` archive is omitted because it can contain credentials and TLS material. The host toolkit and Docker evidence below is the reproduction.

```text
$ nvidia-smi -L
GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)
MIG 2g.0gb Device 0: (UUID: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1)
MIG 1g.0gb Device 1: (UUID: MIG-f177deec-7f6d-5cc3-88d9-e746daca1917)

$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.19.1
commit: 598a9cbb6239c7d05df9c881bad36df7eff34d78

$ nvidia-ctk cdi list
Found 4 CDI devices
nvidia.com/gpu=0
nvidia.com/gpu=all
nvidia.com/pva=0
nvidia.com/pva=all

$ sudo nvidia-ctk cdi generate --device-id=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 --output=/tmp/nvidia-cdi-mig.yaml
Auto-detected mode as 'csv'
failed to generate CDI spec: failed to create device CDI specs: failed to construct device spec generators: unsupported device id: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1

$ sudo nvidia-ctk cdi generate --mode=nvml --output=/tmp/nvidia-cdi-nvml.yaml
Selecting /dev/nvidia-caps/nvidia-cap12 as /dev/nvidia-caps/nvidia-cap12
Selecting /dev/nvidia-caps/nvidia-cap13 as /dev/nvidia-caps/nvidia-cap13
Selecting /dev/nvidia-caps/nvidia-cap21 as /dev/nvidia-caps/nvidia-cap21
Selecting /dev/nvidia-caps/nvidia-cap22 as /dev/nvidia-caps/nvidia-cap22
failed to locate libcuda.so: libcuda.so.595.78: not found
failed to locate libcuda.so: libnvidia-ml.so.595.78: not found

$ docker run --rm --gpus device=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead

$ docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
failed to generate CDI spec for mode "csv": failed to construct device spec generators: unsupported device id: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1

$ docker run --rm --device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1

$ openshell sandbox create --help
--gpu []
Request GPU resources for the sandbox.
Omit COUNT for the driver's default GPU selection, or pass COUNT to request a specific number of GPUs.
# no --gpu-device flag in 0.0.85

$ nemoclaw onboard --help
--sandbox-gpu-device=
OpenShell GPU device selector to pass to sandbox create; requires --sandbox-gpu
# v0.0.103 has no --sandbox-gpu-mig-profile
```

A `libcuda.so.595.78 -> libcuda.so.1` symlink was tested and reverted. Toolkit then failed with `unexpected version detected: 595.78 != 1.1`.

### Logs

```text
Policy fetch failed ... sandbox not found
```

That log appears after `docker compose up` or `docker compose down` unregisters an OpenShell sandbox. It also appears if a compose file keeps orphaned sandbox credentials. It is not evidence that the MIG cap nodes are wrong.

CSV `devices.csv` on this host lists `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, and `/dev/nvmap`. It does not list `/dev/nvidia-caps/*`.

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start with the `nemoclaw onboard --help` and `openshell sandbox create --help` entry points, then reproduce the documented selector and CDI failures on Jetson Thor. Trace how NemoClaw v0.0.103 passes `--sandbox-gpu-device` and `NVIDIA_VISIBLE_DEVICES`; done means GI 1 can be selected without unsupported CSV/CDI paths and the gateway registration survives sandbox lifecycle operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.