NVIDIA / NVIDIA/NemoClaw

[DGX Station][Onboard] DCode onboard leaves a Ready sandbox unregistered when live provider validation fails

Open
#10,877 0 comments 0 reactions 0 assignees View on GitHub
area: onboarding area: sandbox integration: dcode needs: unblock NV QA platform: dgx-station
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

On NemoClaw v0.0.118, a non-interactive DCode onboard on DGX Station creates a sandbox, waits for it to become Ready, and verifies direct GPU/CUDA access successfully. The onboard then fails during live model/provider validation, refuses to update registry metadata, and leaves the Ready sandbox unregistered for manual recovery.

Platform scope: Reproduced on DGX Station only; other platforms not tested for this exact DCode live-validation failure.
Regression: Unknown — earlier versions were not bisected for this exact DCode live-validation path.
OpenShell issue: No — the failing command is the NemoClaw DCode onboard flow after OpenShell sandbox creation reports Ready.

## Environment

```text
Device: DGX Station
OS: Linux 6.17.0-1029-nvidia-64k
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: Docker Engine 29.6.1
OpenShell CLI: openshell 0.0.106
NemoClaw: nemoclaw v0.0.118
OpenClaw: N/A (DCode / LangChain Deep Agents Code sandbox)
```

## Steps to Reproduce

1. On a DGX Station host with NemoClaw v0.0.118 installed, run a non-interactive DCode onboard for a named sandbox.
2. Use the NVIDIA Endpoints provider with model `nvidia/nemotron-3-ultra-550b-a55b`.
3. Allow the onboard flow to create the sandbox and complete the GPU/CUDA readiness checks.
4. Observe the final live model/provider validation and registry publication stage.

## Expected Result

After the sandbox reports Ready and GPU/CUDA probes pass, DCode live model/provider validation should either succeed and publish registry metadata, or fail with a recoverable state that allows the user to retry safely. A Ready sandbox should not be left unregistered without validated registry metadata.

## Actual Result

The sandbox is created and reports Ready. Direct GPU checks pass, including `nvidia-smi` when available, process comm write, and `cuInit` through `libcuda.so.1`. Immediately after that, DCode live model/provider validation fails. NemoClaw refuses to update registry metadata and leaves the sandbox unregistered, preserving it for manual identity-bound recovery.

## Logs

```text
[6/8] Creating sandbox
Direct sandbox GPU enabled; allowing OpenShell GPU policy enrichment.
Creating sandbox 'dcode-reonboard'...
Building sandbox image...
Waiting for sandbox to become ready...
Sandbox reported Ready; waiting for the create ownership handoff to finish.
Waiting for sandbox to become ready...
Verifying direct sandbox GPU access...
✓ GPU proof passed: nvidia-smi when available
✓ GPU proof passed: /proc/{pid}/task/{tid}/comm write
✓ GPU proof passed: cuInit(0) via libcuda.so.1
✓ Sandbox CUDA usability proven (cuInit succeeded).

DCode live model/provider validation failed for sandbox 'dcode-reonboard'. The sandbox still exists, but its live route is unverified and registry metadata was not updated.
A NemoClaw rebuild is unsafe here because no verified registry metadata exists.
NemoClaw left unregistered sandbox 'dcode-reonboard' in place because OpenShell can delete it only by mutable name.
Verify its durable identity before manual cleanup; do not act by name alone.
Then rerun the original `nemoclaw onboard` command.
Sandbox 'dcode-reonboard' was retained after registry publication failed.
```

Contributor guide

Open the contributing guide

Research direction

Start with the non-interactive `nemoclaw onboard` entry point and reproduce the DGX Station flow through the logged stages: sandbox creation, Ready/GPU checks, live provider validation, and registry publication. Trace the failure path after validation fails; done means the Ready sandbox can be retried or recovered safely without being left unregistered or requiring unsafe name-based cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.