NVIDIA / NVIDIA/NemoClaw

[Jetson][Sandbox] Sandbox GPU passthrough proof fails for the non-root sandbox user on JetPack 6.2 IGX Orin — onboarding aborts

Open
#7,610 4 comments 0 reactions 1 assignee Assigned to @cjagwani View on GitHub
area: onboarding area: sandbox platform: jetson v0.0.127 VDR
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

On JetPack 6.2 (L4T R36.5.1) IGX Orin, GPU detection succeeds but the end-to-end sandbox GPU passthrough proof fails for the unprivileged in-sandbox user, so `nemoclaw onboard` aborts at the GPU proof gate. The user can only complete onboarding by forcing CPU behavior with `NEMOCLAW_SANDBOX_GPU=0`.

Isolation shows the passthrough plumbing is correct and the gap is the non-root user: running `cuInit(0)` as root inside the same container returns 0 (success), while the unprivileged sandbox user (even with the `--group-add` video+995 the onboarder applies) cannot initialize CUDA.

**Platform scope:** Reproduced on JetPack 6.2 (R36.5.1) IGX Orin only; other Jetson variants / JetPack versions not tested. `cuInit` as root succeeds, so this may be IGX-Orin-specific device-access strictness.

**Regression:** Unknown — earlier versions not tested.

## Environment

```text
Device: NVIDIA IGX Orin Developer Kit
OS: Ubuntu 22.04 (JetPack 6.2, L4T R36.5.1)
Architecture: aarch64
Node.js: v22.23.1
npm: 10.9.8
Docker: 29.6.2 (installed via official convenience script; nvidia-container-toolkit 1.20.0~rc.1, CSV mode; nvidia runtime configured via `nvidia-ctk runtime configure --runtime=docker`)
OpenShell CLI: 0.0.85
NemoClaw: v0.0.93
OpenClaw: 2026.7.1
```

## Steps to Reproduce

1. Fresh install on IGX Orin JetPack 6.2:
```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```
2. Configure the NVIDIA container runtime for Docker:
```bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
3. Onboard against NVIDIA Endpoints (any cloud model):
```bash
nemoclaw onboard --name jetson-vdr --agent openclaw
```
4. Watch `[6/8] Creating sandbox` → the GPU proof step.

## Expected Result

The sandbox GPU proof passes and onboarding completes with sandbox GPU enabled.

## Actual Result

Detection / preflight pass:

```text
OK NVIDIA GPU detected (Orin (nvgpu), 54790 MB); Sandbox GPU: enabled (auto)
OK Granting sandbox user access to Jetson Tegra GPU device nodes via --group-add 44, 995
OK Docker container mode: --runtime nvidia (NVIDIA_VISIBLE_DEVICES=all)
OK GPU proof passed: nvidia-smi when available
OK GPU proof passed: /proc/{pid}/task/{tid}/comm write
```

But the CUDA proof fails, aborting onboarding:

```text
Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
GPU proof failed inside an executable sandbox
-> throws at docker-gpu-sandbox-create.js verifyGpuOrExit; onboarding does not finish.
```

Isolation (device nodes ARE mounted via CSV: `/dev/nvmap`, `/dev/nvgpu/igpu0/*`, `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvsciipc`, `/dev/nvhost-*`; libcuda present):

```text
- cuInit(0) as ROOT in the container -> 0 (CUDA_SUCCESS)
- cuInit(0) as sandbox user (uid 998, +grp 44,995) -> 801 (CUDA_ERROR_NOT_SUPPORTED)
- onboard proof (sandbox user) -> 999 (NvRmMemInitNvmap Permission denied)
```

Groups alone (video=44 + 995) are insufficient for the non-root user to init CUDA on this IGX Orin.

**Workaround:** `NEMOCLAW_SANDBOX_GPU=0` (CPU) lets onboarding complete.

## Logs

```text
Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
proof_error=Sandbox GPU proof returned failed status: cuInit(0) via libcuda.so.1
```

## Related

- #7209 (Jetson GPU passthrough E2E verification item), PR #5613.
- #4231 (prior "Sandbox GPU enabled but CUDA unusable" status bug, fixed).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.