NVIDIA / NVIDIA/NemoClaw

[Linux][Sandbox] rebuild refuses to restore Deep Agents state with a snapshot acceleration error, so a skill created before the rebuild is lost and rebuild exits 1

Open
#11,803 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

`rebuild` cannot restore a Deep Agents sandbox's application state. The rebuild itself completes and the sandbox returns to Ready, but the restore phase is refused and the sandbox comes back empty, so a user skill created before the rebuild is gone afterwards. `rebuild` exits 1.

The restore is refused with `Cannot restore provider snapshot authority: Runtime snapshot provider failed: sandbox {name} cannot represent the snapshot acceleration state`, and the summary line reports `Partial restore: none` — nothing at all is restored, not merely part of the state.

This makes snapshot-then-rebuild unusable as a state-preservation workflow: the snapshot is created successfully and reported as covering the state, but it can never be applied back by the rebuild that is supposed to consume it. The only recovery offered is a manual restore from the backup directory.

- **Platform scope:** Reproduced on DGX Station only; other platforms not tested this pass. On DGX Spark the rebuild could not be driven as far as the restore phase, because its inference preflight probe kept failing against a rate-limited upstream, so that platform is untested rather than known-good.
- **Regression:** Unknown — earlier builds not tested this pass.
- **OpenShell issue:** No — the gateway and the sandbox are healthy throughout; the sandbox is recreated successfully and reaches Ready. The failure is in NemoClaw's own restore step.

## Environment

```text
Device: NVIDIA DGX Station
OS: Ubuntu 24.04 LTS
Architecture: aarch64
Docker: Docker version 29.6.1
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.124
Deep Agents: LangChain Deep Agents Code v0.1.55
```

## Steps to Reproduce

Use a brand-new sandbox so that no earlier snapshot or backup exists.

1. Onboard a fresh Deep Agents Code sandbox and confirm it is Ready. Confirm it has no snapshots yet:
```bash
nemoclaw {sandbox} snapshot list
```
2. Create a user skill inside the sandbox and confirm it is listed:
```bash
nemoclaw {sandbox} exec -- sh -lc 'dcode skills create report-writer'
nemoclaw {sandbox} exec -- sh -lc 'dcode skills list'
```
3. Take the snapshot:
```bash
nemoclaw {sandbox} snapshot create --name pre-rebuild
```
4. Rebuild:
```bash
nemoclaw {sandbox} rebuild --yes
```
5. List the skills again:
```bash
nemoclaw {sandbox} exec -- sh -lc 'dcode skills list'
nemoclaw {sandbox} exec -- sh -lc 'ls /sandbox/.deepagents/agent/skills/'
```

## Expected Result

Rebuild exits 0, the sandbox returns to Ready, and the state captured by the snapshot is restored, so `dcode skills list` still shows `report-writer` after the rebuild.

## Actual Result

Steps 1–3 behave correctly:

```text
snapshot list before 0 snapshots
dcode skills create skill created
dcode skills list report-writer listed
snapshot create exit 0, "Snapshot v1 name=pre-rebuild created (2 directories, 1 files)"
```

Step 4 rebuilds the sandbox but refuses the restore, and exits 1:

```text
Restoring workspace state...
Restore blocked: Cannot restore provider snapshot authority: Runtime snapshot provider
failed: sandbox '{sandbox}' cannot represent the snapshot acceleration state.
Partial restore: none
Manual restore available from: {backup path}

Sandbox '{sandbox}' rebuilt but some post-restore steps were incomplete
State restore was incomplete - backup available at: {backup path}
State recovery remains incomplete. Correct the restore error, then run
`nemoclaw {sandbox} rebuild` again.
State restore remained incomplete after rebuilding '{sandbox}'.
```

Step 5 shows the state is gone:

```text
dcode skills list report-writer not listed (0 matches)
ls /sandbox/.deepagents/agent/skills/ No such file or directory
status Phase: Ready
```

Reproduced on two independent brand-new sandboxes, each with zero snapshots before the test, with identical output. Re-running rebuild as the message suggests does not help, because the same restore refusal occurs again.

## Related Issues / not duplicate of

No existing report covers the restore phase being refused with a snapshot-acceleration-state error. This is the restore side of `rebuild`, not snapshot creation, which succeeds here.

Contributor guide

Open the contributing guide

Research direction

Start with the `rebuild` restore phase and the snapshot provider authority error, reproducing it with the commands in the issue on a fresh sandbox. Trace why the snapshot acceleration state is rejected after rebuild; done means the rebuild exits 0, returns to Ready, and the created skill remains listed.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, shell, typescript
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.