NVIDIA / NVIDIA/NemoClaw

[Ubuntu 26.04][Onboard] GPU rejection during vLLM/NIM sandbox creation has no compatibility fallback path

Open
#10,669 0 comments 0 reactions 0 assignees View on GitHub
area: local-models area: onboarding area: sandbox NV QA platform: ubuntu
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

Test coverage assumed that when GPU passthrough is rejected before or during sandbox creation, onboard falls back to a digest-pinned compatibility image and completes successfully. Removing the host's NVIDIA CDI spec to simulate this rejection shows the product only has a single, earlier host-level preflight gate (missing CDI spec) that aborts onboarding entirely with remediation steps — there is no deeper "attempt GPU, catch rejection, retry non-GPU" fallback anywhere in the sandbox-creation code path.

Platform scope: Reproduced on Ubuntu 26.04 only; other platforms not tested.
Regression: Unknown — earlier versions not tested; this fallback path may never have existed.

## Environment

```text
Device: Ubuntu 26.04 server, NVIDIA RTX PRO 6000 GPU
OS: Ubuntu 26.04
Architecture: x86_64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.7.2
OpenShell CLI: 0.0.106
NemoClaw: 0.0.116
OpenClaw: 2026.7.1
```

## Steps to Reproduce

1. On a host with a real NVIDIA GPU, temporarily move aside the CDI spec files (`/etc/cdi/nvidia.yaml`, `/run/cdi/nvidia.yaml`) to simulate GPU passthrough becoming unavailable mid-onboard.
2. Run:
```bash
nemoclaw onboard --agent openclaw --name {sandbox} --non-interactive --fresh --yes --yes-i-accept-third-party-software
```
(no `--no-gpu` flag, so GPU passthrough is attempted)
3. Observe the onboarding result.
4. Restore the CDI spec files afterward.

## Expected Result

GPU rejection during or immediately before sandbox creation triggers one automatic compatibility fallback: onboarding completes via a non-GPU, digest-pinned managed image, with no sandbox-creation error surfaced to the user.

## Actual Result

```text
[1/8] Preflight checks
──────────────────────────────────────────────────
✗ Docker is configured for CDI device injection (CDISpecDirs is set), but the NVIDIA GPU CDI spec is missing or stale. OpenShell GPU startup can fail until the CDI spec is refreshed.

Suggested fix:

- Generate NVIDIA CDI device specs (generate_nvidia_cdi_spec): Docker is configured for CDI device injection (CDISpecDirs is set) but no nvidia.com/gpu CDI spec is present on the host. A GPU-enabled gateway start can fail until the CDI spec is refreshed and verified.
sudo mkdir -p '/etc/cdi'
sudo systemctl enable --now nvidia-cdi-refresh.path nvidia-cdi-refresh.service
sudo systemctl start nvidia-cdi-refresh.service
nvidia-ctk cdi list # verify nvidia.com/gpu entries appear
sudo nvidia-ctk cdi generate --output='/etc/cdi/nvidia.yaml' # fallback if the refresh service does not repair the spec
nvidia-ctk cdi list # verify nvidia.com/gpu entries appear
nemoclaw onboard # or rerun with --no-gpu to skip GPU passthrough

Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume --name {sandbox}
Completed steps are skipped; pass --fresh instead to start over.
```

## Logs

Not captured beyond the console output shown above.

Contributor guide

Open the contributing guide

Research direction

Start with the `nemoclaw onboard --agent openclaw ...` reproduction after moving the NVIDIA CDI spec files aside. Trace the onboarding preflight and sandbox-creation paths to determine where GPU rejection is handled. Done means one non-GPU retry uses a digest-pinned compatibility image and onboarding completes without surfacing a sandbox-creation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, node.js, typescript
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.