NVIDIA / NVIDIA/NemoClaw

[DGX Station][Upgrade] strict pre-upgrade backup counts an uncapturable sandbox as neither failed nor skipped, so the gate never fires

Open
#11,795 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

A registered sandbox that cannot be captured at all is counted as neither failed nor skipped by the strict pre-upgrade backup, so the strict gate does not fire. The summary line reports a clean backup while one sandbox was never backed up, and the installer continues past the backup phase instead of stopping before the legacy OpenShell gateway work.

An operator reading the summary would reasonably conclude every sandbox was captured. The sandbox that was not captured is mentioned only on a separate line, in different wording, and is not reflected in any of the three counters.

**Platform scope:** reproduced on DGX Station GB300 (Linux aarch64); the counting happens in the installer's backup phase and is not expected to be hardware specific.
**Regression:** Unknown — earlier builds not tested for this exact counting.
**OpenShell issue:** No — the miscount and the missing abort are on the NemoClaw installer side; the OpenShell gateway was correctly left at its old version.

## Environment

```text
Device: NVIDIA DGX Station GB300
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Starting from: nemoclaw v0.0.121, openshell 0.0.106 (gateway still needing the legacy upgrade)
Upgrading to: the current maintained release via the documented installer
```

## Steps to Reproduce

1. On a host running an older maintained release whose OpenShell gateway still needs the legacy upgrade, register two managed-image sandboxes:
- `keep-good` — Ready, its pre-upgrade capture will succeed
- `keep-stuck` — registered, but made uncapturable
2. Make `keep-stuck` uncapturable by removing its backing container out of band, so it can no longer be started for backup: `docker rm -f `
3. Confirm both are still registered: `nemoclaw list`
4. Run the documented in-place upgrade: `curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash`
5. Read the pre-upgrade backup phase: the per-sandbox result lines, the summary counters, and the line printed when the installer stops.

## Expected Result

The sandbox that could not be captured is reported as failed or skipped and is counted as such, for example `Pre-upgrade backup: 1 backed up, 1 failed` or `... 1 skipped`, and the strict gate fires before any legacy gateway work, printing the documented abort line.

## Actual Result

The backup phase prints:

```text
Backing up 'keep-good'...
keep-good: 12 dirs, 1 files, stored under the gateway's rebuild-backups directory
Pre-upgrade backup: 1 backed up, 0 failed, 0 skipped
Backups stored in:
1 recorded sandbox(es) were not found on their recorded gateway: keep-stuck.
```

So `keep-stuck` is counted in none of the three counters, and the summary reads as a clean backup. The installer does not stop here — it proceeds into the sandbox upgrade phase:

```text
keep-good v2026.7.1 unchanged; NemoClaw image v0.0.121 to v0.0.123 (running)
keep-stuck v2026.7.1 unchanged; NemoClaw image v0.0.121 to v0.0.123 (stopped)
Backup recovery blocked:
keep-stuck no validated pre-upgrade backup was found
```

and only much later exits nonzero with a different message:

```text
Installation incomplete: one or more existing sandboxes failed to upgrade.
```

The documented abort line about stopping before retiring the legacy OpenShell gateway never appears.

**What did work:** the legacy gateway was not retired (openshell stayed at 0.0.106), both sandboxes remained registered, and `keep-good` status stayed Ready. So the end state is safe; the defect is that the strict backup gate did not fire and its counters do not account for an uncapturable sandbox.

## Related — checked and distinct

- **stopped sandboxes are started for backup but remain SSH-unreachable and block strict upgrade. That is the opposite outcome: there the gate fires and blocks; here a sandbox that cannot be captured at all does not make the gate fire. The conditions differ too — there the backing container still exists and only SSH is unreachable; here the backing container was removed so the sandbox cannot be started for backup at all. Worth fixing together, since both concern which uncapturable states the strict gate recognises.

Contributor guide

Open the contributing guide

Research direction

Start at the documented installer's pre-upgrade backup phase and reproduce the case with a registered sandbox whose backing container is removed using `docker rm -f`. Trace how the per-sandbox result reaches the summary counters and strict gate. Done means the uncapturable sandbox is counted as failed or skipped and the documented abort occurs before legacy gateway work.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.