[Ubuntu 24.04][Sandbox] rebuild --force aborts and leaves Phase Error when the exec relay is unavailable, the one case it exists for
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
`nemoclaw rebuild --force` aborts and leaves the sandbox in Phase Error when the sandbox's exec relay is unavailable, which is the one situation the `--force` path exists to recover from. The command refuses because it cannot back up sandbox state first, and the state backup runs through the very relay that is already gone, so `--force` and "back up before rebuilding" are mutually exclusive in exactly this case.
A two-arm experiment on one host isolates it: the same command against a healthy sandbox on the same host, same gateway and same release exits 0, completes its backup and returns the sandbox to Ready. Only the unhealthy arm fails.
The operator is left worse off than before running the documented recovery: the sandbox was merely stopped beforehand, and afterwards it reports Phase Error with an incomplete snapshot retained that is explicitly excluded from restore selection.
- **Platform scope:** Reproduced on Ubuntu 24.04 x86_64; other platforms not tested for this path.
- **Regression:** Unknown — earlier releases were not tested for the `--force` variant of this path.
- **OpenShell issue:** No. The aborting command is NemoClaw's own rebuild.
## Environment
```text
Device: Ubuntu 24.04 x86_64 workstation
OS: Ubuntu 24.04.4 LTS
Architecture: x86_64
Node.js: v22.23.1
Docker: Docker version 29.5.2
OpenShell CLI: openshell 0.0.106
NemoClaw: nemoclaw v0.0.120
OpenClaw: 2026.7.1
```
## Steps to Reproduce
1. Onboard a sandbox on an isolated gateway port and confirm it reaches Ready:
```bash
nemoclaw onboard --agent openclaw --name {sandbox} --fresh --yes --no-gpu
nemoclaw {sandbox} status
```
2. Make the exec relay unavailable by stopping the sandbox container:
```bash
docker stop {container id labelled openshell.ai/sandbox-name={sandbox}}
```
3. Confirm the relay really is gone, so the precondition is established rather than assumed:
```bash
nemoclaw {sandbox} exec -- true
```
This exits non-zero.
4. Run the documented forced recovery and record the exit code:
```bash
nemoclaw {sandbox} rebuild --force --yes
```
5. Ask for the sandbox state:
```bash
nemoclaw {sandbox} status
```
6. Control arm, same host, same gateway, minutes apart: onboard a second sandbox, leave it healthy, and run the same command against it.
## Expected Result
Step 4 completes and returns the sandbox to Ready. If state cannot be backed up because the relay is unavailable, `--force` is the flag that is supposed to proceed anyway, so it should either recover through the host-side path or state plainly that no forced recovery is possible for this condition. Step 5 should not report Error.
## Actual Result
Step 4 exits 1. Every state directory fails to back up, not a subset:
```text
Backing up sandbox state...Failed to back up sandbox state.
Inspect the per-directory failure reasons below along with the mount's ownership and permissions.
Failed: agents, extensions, workspace, skills, hooks, canvas, cron, memory, telegram, wechat,
whatsapp, credentials
Failed files: openclaw.json
Incomplete snapshot retained for manual recovery:
{home}/.nemoclaw/gateways/{port}/rebuild-backups/{sandbox}/{timestamp}
It is excluded from snapshot restore selection.
Aborting rebuild to prevent data loss.
```
Step 5 reports `Phase: Error`.
Step 6, the control on the same host, exits 0. Its backup succeeds, the rebuild completes and the control sandbox reports `Phase: Ready`. The host is therefore capable of a successful forced rebuild; only the arm whose relay is unavailable fails.
The failure of every directory at once, rather than a subset, is consistent with the backup path itself running through the relay that step 2 removed.
## Logs
```text
Experimental arm, relay unavailable:
exec -- true exit 1 (precondition confirmed)
rebuild --force --yes exit 1
status Phase: Error
Control arm, healthy sandbox, same host and gateway:
rebuild --force --yes exit 0
Backup: {home}/.nemoclaw/gateways/{port}/rebuild-backups/{other sandbox}/{timestamp}
status Phase: Ready
```
Contributor guide
Research direction
Start at the `nemoclaw rebuild --force --yes` CLI entry point and trace the state-backup call used before rebuilding. Reproduce the stopped-container sequence and compare it with the healthy control arm. Done means forced recovery either reaches Ready without the relay or clearly reports that recovery is unsupported, without leaving Phase Error or using the incomplete snapshot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux, typescript
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100