NVIDIA / NVIDIA/NemoClaw

[Linux][Upgrade] stopped sandboxes are started for backup but remain SSH-unreachable and block strict upgrade

Open
#11,936 0 comments 0 reactions 0 assignees View on GitHub
area: install area: sandbox NV QA platform: ubuntu
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

On NemoClaw v0.0.126, a strict pre-upgrade backup fails when the inventory contains stopped sandboxes. The installer starts each stopped sandbox, attempts the backup, returns it to the stopped state, and then reports that its in-sandbox SSH endpoint did not answer. The strict backup policy consequently aborts the upgrade.

Platform scope: Reproduced on Ubuntu 24.04 x86_64 in v0.0.126. The same behavior was previously reproduced on Ubuntu 22.04 and Ubuntu 26.04 GPU in v0.0.124.

Regression: This is the stopped-sandbox backup behavior previously addressed by #6500 / #6723, but it remains reproducible in current releases.

OpenShell issue: No evidence. NemoClaw owns the pre-upgrade backup orchestration and readiness timing.

## Environment

```text
NemoClaw: v0.0.126
Platform: Ubuntu 24.04 x86_64
Scenario: two stopped registered sandboxes and one healthy sandbox
```

## Steps to Reproduce

1. Create multiple managed sandboxes.
2. Stop two sandboxes and leave one sandbox running and healthy.
3. Run the supported NemoClaw upgrade installer.
4. Observe pre-upgrade backup processing.

## Expected Result

Stopped sandboxes are either started into a backup-ready state and backed up successfully, or handled according to a documented stopped-sandbox policy without being misreported as running-but-SSH-unreachable.

## Actual Result

Each stopped sandbox is started, its backup fails because the in-sandbox SSH endpoint does not answer, and it is returned to the stopped state. The healthy sandbox is backed up successfully. The installer reports:

```text
Pre-upgrade backup: 1 backed up, 2 failed
```

The strict pre-upgrade backup then aborts the upgrade.

## Reproducibility

Reproduced in a clean targeted v0.0.126 run. The sandbox state restoration assertion passes, while the strict backup success assertion fails consistently.

## Related Issues

- #6500 documents the earlier stopped-sandbox strict-backup failure.
- #6723 implemented the prior fix.

Contributor guide

Open the contributing guide

Research direction

Start at the supported NemoClaw upgrade installer and its pre-upgrade backup processing, using the stopped-sandbox reproduction with two stopped sandboxes and one healthy sandbox. Compare the current behavior with the prior fixes referenced in #6500 and #6723. Done means stopped sandboxes are backed up successfully or handled by a documented policy, strict upgrades no longer fail incorrectly, and sandbox state restoration still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
infrastructure, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.