[Jetson Thor][Onboard] Documented GPU-device and CDI paths cannot pin a sandbox to MIG GI 1
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
### Investigation Summary
- On Jetson AGX Thor with MIG enabled, `nvidia-ctk cdi list` reports only `nvidia.com/gpu=0`, `nvidia.com/gpu=all`, `nvidia.com/pva=0`, and `nvidia.com/pva=all`. It does not list MIG UUIDs or `/dev/nvidia-caps` nodes.
- CSV-mode `nvidia-ctk cdi generate --device-id=` fails with `unsupported device id`. `--mode=nvml` selects `nvidia-cap12`/`nvidia-cap13` for GI 1, then fails because it cannot locate `libcuda.so.595.78`.
- `docker run --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=` fails with the same CSV `unsupported device id`. `--device nvidia.com/gpu=` is unresolvable. `--gpus` is rejected on this Jetson host (the runtime requires `--runtime=nvidia`).
- Installed NemoClaw v0.0.103 Jetson onboard uses the NVIDIA runtime and `NVIDIA_VISIBLE_DEVICES=`, so it hits that CSV path. OpenShell 0.0.85 `sandbox create` accepts `--gpu [COUNT]` only. It has no `--gpu-device`.
- A working pin is an unscoped OpenShell create, then a clone of that container with raw `--device` nodes for GI 1 (`nvidia-cap12` + `nvidia-cap13`). `docker compose up`/`down` on that sandbox unregisters it (`Policy fetch failed ... sandbox not found`). That crash is not the MIG device failure.
### Description
NemoClaw and OpenShell cannot pin a sandbox to Jetson Thor MIG GPU instance 1 (`2g.0gb+gfx`) through the documented GPU-device, CDI, or `NVIDIA_VISIBLE_DEVICES` path.
**Expected:** An operator can pin the sandbox to GI 1 and leave GI 2 (`1g.0gb+me`) unused. Documented selectors (`--sandbox-gpu-device`, CDI `nvidia.com/gpu=`, or `NVIDIA_VISIBLE_DEVICES=`) should either attach that partition or fail with a Jetson-specific error that names a supported pin method.
**Actual:** Every documented selector fails in NVIDIA Container Toolkit CSV mode. The Jetson compatibility path in NemoClaw v0.0.103 still sets `NVIDIA_VISIBLE_DEVICES` and therefore fails the same way. Public docs still tell operators to pass `--sandbox-gpu --sandbox-gpu-device ` into `openshell sandbox create`. OpenShell 0.0.85 does not implement `--gpu-device`.
**Working operator pin (do not treat this as a supported product interface):**
1. Create the sandbox with GPU off or unscoped (`openshell sandbox create` without a MIG UUID). Keep the gateway registration.
2. Clone that container in place with raw `--device` nodes for GI 1: `/dev/nvidia-caps/nvidia-cap12` (GI) and `/dev/nvidia-caps/nvidia-cap13` (CI 0), plus `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, and `/dev/nvmap` as needed.
3. Do not set `NVIDIA_VISIBLE_DEVICES`. Do not inject CDI. Isolation is the device cgroup.
**Do not recommend these on this JetPack / toolkit combination:**
- `--sandbox-gpu-device` with a MIG UUID or CDI name.
- `NVIDIA_VISIBLE_DEVICES=`.
- `nvidia-ctk cdi generate` without `--output=/tmp/...`. That rewrite of host CDI still emits only `0`/`all`.
- `libcuda.so.595.78` or `libnvidia-ml.so.595.78` symlinks. Toolkit then reports `unexpected version 595.78 != 1.1`.
**Related lifecycle failure (same investigation, different cause):** `docker compose up` or `docker compose down` against an OpenShell-managed sandbox unregisters it with the gateway. The sandbox then crash-loops with `Policy fetch failed ... sandbox not found` even when the MIG device nodes are correct. Orphaned credentials in a compose file reproduce that crash. Prefer `docker restart` or an in-place clone. Do not recreate the sandbox through Compose.
**Proposed product fix:**
- After an unscoped OpenShell create, pin GI 1 with a raw-devices clone (for example `--sandbox-gpu-mig-profile 2g.0gb+gfx`).
- Do not send MIG UUIDs through `--sandbox-gpu-device`, CDI, or `NVIDIA_VISIBLE_DEVICES`.
- Persist the pin without a Compose recreate so the gateway registration stays valid.
- v0.0.103 does not ship `--sandbox-gpu-mig-profile`. A later unreleased source change adds that flag. This issue tracks the released onboard gap.
**Related issues (not duplicates):**
- NVIDIA/NemoClaw#7610: IGX Orin non-root `cuInit` proof. Different platform and failure.
- NVIDIA/NemoClaw#8500: Jetson AGX Thor Express installer epic. GPU access on Thor is in that scope.
- NVIDIA/NemoClaw#3710: k3s nvidia-device-plugin CDI on older Jetson onboard. This host uses the Docker compatibility path.
- NVIDIA/OpenShell#1444: GPU sandbox / `--gpu-device` design. OpenShell 0.0.85 still has `--gpu [COUNT]` only.
- NVIDIA Container Toolkit CSV generation accepts device ids `all` and `0` only. Mixed CSV mode states that MIG ids are not supported. NVML generate on this L4T tree fails on `libcuda.so.1.1` versus `libcuda.so.595.78`.
### Reproduction Steps
1. On Jetson AGX Thor with JetPack L4T R39.2, enable MIG and create the two recommended GPU instances (profile 83 `2g.0gb+gfx` as GI 1, profile 78 `1g.0gb+me` as GI 2), then create default compute instances.
2. Confirm host inventory:
```bash
nvidia-smi -L
nvidia-ctk --version
nvidia-ctk cdi list
ls /dev/nvidia-caps
```
3. Try the documented CDI generate path with output under `/tmp` only:
```bash
sudo nvidia-ctk cdi generate --device-id=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 --output=/tmp/nvidia-cdi-mig.yaml
sudo nvidia-ctk cdi generate --device-id=0:1 --output=/tmp/nvidia-cdi-0-1.yaml
sudo nvidia-ctk cdi generate --mode=nvml --output=/tmp/nvidia-cdi-nvml.yaml
```
4. Try Docker injection:
```bash
docker run --rm --gpus device=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
docker run --rm --device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
```
5. Try the documented NemoClaw selector (v0.0.103):
```bash
nemoclaw onboard --name my-assistant --sandbox-gpu --sandbox-gpu-device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1
```
6. Compare OpenShell create flags:
```bash
openshell sandbox create --help
```
7. Optional lifecycle check: after a gateway-registered unscoped sandbox exists, run `docker compose up` or `docker compose down` against that container. The gateway then reports the sandbox as missing.
### Environment
- OS: Ubuntu 24.04.4 LTS (Noble), aarch64
- Hardware: NVIDIA Jetson AGX Thor, GPU 0 UUID `GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600`
- JetPack / L4T: R39.2.0 (GCID 45755727), kernel 6.8.12-1021-tegra
- Driver userspace: host `libcuda.so.1.1` under `/opt/nvidia/l4t-gpu-libs/openrm/` (no `libcuda.so.595.78` file)
- MIG GI 1: profile 83 `2g.0gb+gfx`, placement 0:2, UUID `MIG-314bde5a-0c51-524c-b404-7fe9c0231de1`, caps `/dev/nvidia-caps/nvidia-cap12` (GI) + `nvidia-cap13` (CI 0)
- MIG GI 2: profile 78 `1g.0gb+me`, placement 2:1, UUID `MIG-f177deec-7f6d-5cc3-88d9-e746daca1917`, caps `nvidia-cap21` + `nvidia-cap22` (left unused)
- Node.js: v22.23.2
- Docker: Docker Engine 29.7.2
- NVIDIA Container Toolkit CLI: 1.19.1, commit `598a9cbb6239c7d05df9c881bad36df7eff34d78`
- OpenShell: 0.0.85
- NemoClaw: v0.0.103 (`nemoclaw --version`)
- CDI: `/var/run/cdi/nvidia.yaml` devices `0` and `all` only; `/etc/cdi` contains `nvidia-pva.yaml` only
- CSV `devices.csv`: includes `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvmap`; does not include `/dev/nvidia-caps/*`
### Debug Output
The `nemoclaw debug` archive is omitted because it can contain credentials and TLS material. The host toolkit and Docker evidence below is the reproduction.
```text
$ nvidia-smi -L
GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)
MIG 2g.0gb Device 0: (UUID: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1)
MIG 1g.0gb Device 1: (UUID: MIG-f177deec-7f6d-5cc3-88d9-e746daca1917)
$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.19.1
commit: 598a9cbb6239c7d05df9c881bad36df7eff34d78
$ nvidia-ctk cdi list
Found 4 CDI devices
nvidia.com/gpu=0
nvidia.com/gpu=all
nvidia.com/pva=0
nvidia.com/pva=all
$ sudo nvidia-ctk cdi generate --device-id=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 --output=/tmp/nvidia-cdi-mig.yaml
Auto-detected mode as 'csv'
failed to generate CDI spec: failed to create device CDI specs: failed to construct device spec generators: unsupported device id: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1
$ sudo nvidia-ctk cdi generate --mode=nvml --output=/tmp/nvidia-cdi-nvml.yaml
Selecting /dev/nvidia-caps/nvidia-cap12 as /dev/nvidia-caps/nvidia-cap12
Selecting /dev/nvidia-caps/nvidia-cap13 as /dev/nvidia-caps/nvidia-cap13
Selecting /dev/nvidia-caps/nvidia-cap21 as /dev/nvidia-caps/nvidia-cap21
Selecting /dev/nvidia-caps/nvidia-cap22 as /dev/nvidia-caps/nvidia-cap22
failed to locate libcuda.so: libcuda.so.595.78: not found
failed to locate libcuda.so: libnvidia-ml.so.595.78: not found
$ docker run --rm --gpus device=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead
$ docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
failed to generate CDI spec for mode "csv": failed to construct device spec generators: unsupported device id: MIG-314bde5a-0c51-524c-b404-7fe9c0231de1
$ docker run --rm --device nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1 ubuntu:24.04 true
CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=MIG-314bde5a-0c51-524c-b404-7fe9c0231de1
$ openshell sandbox create --help
--gpu []
Request GPU resources for the sandbox.
Omit COUNT for the driver's default GPU selection, or pass COUNT to request a specific number of GPUs.
# no --gpu-device flag in 0.0.85
$ nemoclaw onboard --help
--sandbox-gpu-device=
OpenShell GPU device selector to pass to sandbox create; requires --sandbox-gpu
# v0.0.103 has no --sandbox-gpu-mig-profile
```
A `libcuda.so.595.78 -> libcuda.so.1` symlink was tested and reverted. Toolkit then failed with `unexpected version detected: 595.78 != 1.1`.
### Logs
```text
Policy fetch failed ... sandbox not found
```
That log appears after `docker compose up` or `docker compose down` unregisters an OpenShell sandbox. It also appears if a compose file keeps orphaned sandbox credentials. It is not evidence that the MIG cap nodes are wrong.
CSV `devices.csv` on this host lists `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, and `/dev/nvmap`. It does not list `/dev/nvidia-caps/*`.
### Checklist
- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate
Contributor guide
Research direction
Start with the `nemoclaw onboard --help` and `openshell sandbox create --help` entry points, then reproduce the documented selector and CDI failures on Jetson Thor. Trace how NemoClaw v0.0.103 passes `--sandbox-gpu-device` and `NVIDIA_VISIBLE_DEVICES`; done means GI 1 can be selected without unsupported CSV/CDI paths and the gateway registration survives sandbox lifecycle operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100