[Ubuntu 26.04][Onboard] GPU rejection during vLLM/NIM sandbox creation has no compatibility fallback path
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
Test coverage assumed that when GPU passthrough is rejected before or during sandbox creation, onboard falls back to a digest-pinned compatibility image and completes successfully. Removing the host's NVIDIA CDI spec to simulate this rejection shows the product only has a single, earlier host-level preflight gate (missing CDI spec) that aborts onboarding entirely with remediation steps — there is no deeper "attempt GPU, catch rejection, retry non-GPU" fallback anywhere in the sandbox-creation code path.
Platform scope: Reproduced on Ubuntu 26.04 only; other platforms not tested.
Regression: Unknown — earlier versions not tested; this fallback path may never have existed.
## Environment
```text
Device: Ubuntu 26.04 server, NVIDIA RTX PRO 6000 GPU
OS: Ubuntu 26.04
Architecture: x86_64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.7.2
OpenShell CLI: 0.0.106
NemoClaw: 0.0.116
OpenClaw: 2026.7.1
```
## Steps to Reproduce
1. On a host with a real NVIDIA GPU, temporarily move aside the CDI spec files (`/etc/cdi/nvidia.yaml`, `/run/cdi/nvidia.yaml`) to simulate GPU passthrough becoming unavailable mid-onboard.
2. Run:
```bash
nemoclaw onboard --agent openclaw --name {sandbox} --non-interactive --fresh --yes --yes-i-accept-third-party-software
```
(no `--no-gpu` flag, so GPU passthrough is attempted)
3. Observe the onboarding result.
4. Restore the CDI spec files afterward.
## Expected Result
GPU rejection during or immediately before sandbox creation triggers one automatic compatibility fallback: onboarding completes via a non-GPU, digest-pinned managed image, with no sandbox-creation error surfaced to the user.
## Actual Result
```text
[1/8] Preflight checks
──────────────────────────────────────────────────
✗ Docker is configured for CDI device injection (CDISpecDirs is set), but the NVIDIA GPU CDI spec is missing or stale. OpenShell GPU startup can fail until the CDI spec is refreshed.
Suggested fix:
- Generate NVIDIA CDI device specs (generate_nvidia_cdi_spec): Docker is configured for CDI device injection (CDISpecDirs is set) but no nvidia.com/gpu CDI spec is present on the host. A GPU-enabled gateway start can fail until the CDI spec is refreshed and verified.
sudo mkdir -p '/etc/cdi'
sudo systemctl enable --now nvidia-cdi-refresh.path nvidia-cdi-refresh.service
sudo systemctl start nvidia-cdi-refresh.service
nvidia-ctk cdi list # verify nvidia.com/gpu entries appear
sudo nvidia-ctk cdi generate --output='/etc/cdi/nvidia.yaml' # fallback if the refresh service does not repair the spec
nvidia-ctk cdi list # verify nvidia.com/gpu entries appear
nemoclaw onboard # or rerun with --no-gpu to skip GPU passthrough
Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume --name {sandbox}
Completed steps are skipped; pass --fresh instead to start over.
```
## Logs
Not captured beyond the console output shown above.
Contributor guide
Research direction
Start with the `nemoclaw onboard --agent openclaw ...` reproduction after moving the NVIDIA CDI spec files aside. Trace the onboarding preflight and sandbox-creation paths to determine where GPU rejection is handled. Done means one non-GPU retry uses a digest-pinned compatibility image and onboarding completes without surfacing a sandbox-creation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, node.js, typescript
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100