NVIDIA / NVIDIA/NemoClaw

[DGX Station][Upgrade] backup calls a running sandbox not running, so the skip override never applies

Open
#11,899 0 comments 0 reactions 0 assignees View on GitHub
needs: triage NV QA
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

The strict pre-upgrade backup reports a sandbox that is genuinely running as "not running". Because of that, the documented opt-in that is supposed to let the upgrade continue past an unreachable sandbox never applies, and setting it changes nothing at all.

The product contradicts itself on one unchanged state. Against the same three sandboxes, at the same moment, the standalone backup command correctly reports one running sandbox whose in-sandbox SSH endpoint did not answer, and offers the opt-in by name. The installer, run seconds later with nothing changed, calls the same sandbox "not running" and tells the user to start it — a sandbox that is already running and healthy.

Two consequences for the user:

- **The advice is impossible to act on.** There is nothing to start. The sandbox container is up and healthy, and the gateway lists it Ready. Following the instruction cannot clear the condition.
- **The documented escape hatch is dead.** The opt-in only covers sandboxes classified as unreachable, and this misclassification keeps the sandbox out of that group. Running the installer with the opt-in set produces output byte-identical to running it without, and the upgrade still stops.

Separate but from the same runs, recorded here rather than as its own report: a failed backup leaves an empty timestamped backup directory behind. 231 of them accumulated over this session.

- **Platform scope:** Reproduced on DGX Station GB300 (Ubuntu 24.04, aarch64) only; other platforms not tested.
- **Regression:** Unknown — earlier versions not tested for this path.
- **OpenShell issue:** No

## Environment

```text
Device: NVIDIA DGX Station GB300
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: Docker version 29.6.1, build 8900f1d
OpenShell CLI: 0.0.116
NemoClaw: v0.0.126
OpenClaw: 2026.7.1
```

## Steps to Reproduce

1. On a host running the current release, create three sandboxes and confirm all three reach Ready.

2. Make exactly one of them unreachable over SSH **without stopping it**. A host-level stop signal to that container's supervisor process works and is reversible.

Note for anyone reproducing this: pausing the container, and disconnecting it from its network, both fail to create this state — the product reads either one as not running, which is a different path.

3. Confirm the state before going further:

```bash
docker ps
openshell sandbox list
```

The container must show as running and healthy, and the gateway must list all three sandboxes Ready.

4. Confirm how the standalone backup command classifies it:

```bash
nemoclaw backup-all
```

5. Now run the documented in-place upgrade and compare:

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

6. Run it again with the documented opt-in set, and compare the output to step 5 line by line:

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 bash
```

7. Revert the host-level stop signal and confirm the sandbox backs up normally again.

## Expected Result

A running sandbox whose SSH endpoint does not answer is reported as unreachable, not as stopped, and the message names the SSH transport failure and the opt-in that can get the user past it — the same way the standalone backup command already does.

With the opt-in set, that sandbox is classified unreachable and skipped, the upgrade continues past the backup phase and completes, and the skipped sandbox can be recovered afterwards from its latest validated backup.

A backup that fails leaves no directory behind.

## Actual Result

Precondition, verified at the moment of the run:

```text
docker ps the sandbox container is running, Up and healthy
(still running immediately after the installer exited)
openshell sandbox list all three sandboxes Ready
the supervisor process stopped state
```

Step 4, the standalone backup command on that exact state, verbatim:

```text
sandbox-c: backup failed (agents, extensions, workspace, skills, hooks, canvas, cron,
memory, credentials, openclaw.json)
Pre-upgrade backup: 2 backed up, 1 failed, 0 skipped
1 running sandbox(es) could not be backed up because their in-sandbox SSH endpoint did
not answer.
To upgrade now and recover them afterwards from their latest validated backup, re-run
with NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1.
```

Correct on every point: running, SSH endpoint, the opt-in, the recovery path.

Step 5, the installer on the same unchanged state, verbatim:

```text
Backing up 'sandbox-b'...
sandbox-b: 12 dirs, 1 files
Skipping 'sandbox-c' (not running; start the sandbox/container and rerun
'nemoclaw backup-all' so NemoClaw can capture a fresh snapshot)
Backing up 'sandbox-a'...
sandbox-a: 12 dirs, 1 files

Pre-upgrade backup: 2 backed up, 0 failed, 1 skipped

Strict pre-upgrade backup requires every registered sandbox to be backed up;
1 sandbox(es) were skipped.
1 skipped sandbox(es) were not running. Start each sandbox/container, then rerun the
installer or 'nemoclaw backup-all'.
Resolve each skipped sandbox using its reason above and retry.
[ERROR] Pre-upgrade backup stopped the installer.
exit 1
```

The SSH transport failure is never mentioned, the opt-in is never printed, and the recovery path is never printed. Reproduced three times.

Step 6, with the opt-in set, verbatim:

```text
Skipping 'sandbox-c' (not running; ...)
Pre-upgrade backup: 2 backed up, 0 failed, 1 skipped
Strict pre-upgrade backup requires every registered sandbox to be backed up;
1 sandbox(es) were skipped.
[ERROR] Pre-upgrade backup stopped the installer.
exit 1
```

Byte-identical to step 5. The opt-in has no observable effect.

Step 7, after reverting the stop signal, the same sandbox backs up normally, confirming nothing was wrong with it other than being unreachable.

The empty-directory part: the failed backup in step 5 created a timestamped directory under the sandbox's backup folder containing zero entries.

Why this is not a local environment problem: the product disagrees with itself about one unchanged state. No choice of how the sandbox was made unreachable can cause the standalone backup command and the installer to classify the same sandbox differently at the same moment.

## Related Bugs / not duplicate of

#11795 — a sandbox that cannot be captured is counted in none of the three counters, so the strict gate never fires at all. That is not this report: here the counters are correct and the gate does fire. The defect is the reason it gives and the dead opt-in that follows from it.

There is also existing coverage of the opposite misclassification: stopped sandboxes that are started for backup and then reported as running-but-SSH-unreachable. This report is the mirror image — a genuinely unreachable running sandbox reported as stopped. The two are worth fixing together, since both concern how the backup phase decides which bucket a sandbox belongs in.

## Logs

```text
Not captured beyond the terminal output quoted in Actual Result.
```

Contributor guide

Open the contributing guide

Research direction

Start by running the documented nemoclaw backup-all command and the installer from nemoclaw.sh against the reproduced running-but-SSH-unreachable sandbox state. Trace the backup classification and the NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP path, then inspect how failed backups create timestamped directories. Done means both commands classify the state consistently, the opt-in permits the upgrade, and failed backups leave no empty directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, node.js, typescript
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.