NVIDIA / NVIDIA/NemoClaw

[DGX Station][Sandbox] container stop/start wedges the sandbox in Error; neither start nor recover can recover it

Open
#11,885 0 comments 0 reactions 0 assignees View on GitHub
needs: triage NV QA UAT
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

A container-level interruption wedges a sandbox in phase Error, and neither
`start` nor `recover` can bring it back. Stopping and starting the sandbox's own
container through the container runtime - the most ordinary interruption a user
or a host reboot produces - leaves the sandbox permanently unusable even though
the container itself is running and healthy.

`nemoclaw {sandbox} start` then polls for 60 seconds, reports only "Status: Error"
on every poll, and exits 1. `nemoclaw {sandbox} recover` exits 1 as well, and so does
`nemoclaw {sandbox} connect --probe-only`. There is no remaining command that
returns the sandbox to Ready; only destroy and re-onboard clears it.

The same dead end is reached from a Docker daemon restart. The sandbox container
is created with restart policy "no", so restarting the daemon leaves the
container exited and the sandbox in Error; `start` then refuses outright with
"sandbox must be Stopped to start (current phase: Error)". A host reboot puts a
user in exactly this state.

This is NOT the already-fixed stop/start defect: a control run on the same host,
same build, same sandbox confirms the supported `nemoclaw {sandbox} stop` followed
by `nemoclaw {sandbox} start` still works correctly (exit 0, back to Ready in 37s,
probe result=ready). Only the container-level interruption path is broken.

Platform scope: Reproduced on two independent DGX Station hosts; other platforms
not tested this pass.
Regression: Unknown - earlier builds not tested this pass.
OpenShell issue: No - the OpenShell gateway answers throughout and the container
runtime reports the container Up and healthy. The failure is in NemoClaw's own
start/recover handling of the Error phase.

## Environment

```text
Device: NVIDIA DGX Station GB300 (two independent hosts)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Kernel: 6.17.0-1029-nvidia-64k
Node.js: v22.23.2
npm: 10.9.8
Docker: Docker version 29.6.1, build 8900f1d
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.125
OpenClaw: 2026.7.1 (2d2ddc4)
```

## Steps to Reproduce

```bash
Use a freshly onboarded sandbox so no earlier state is involved.

1. Onboard a fresh OpenClaw sandbox and confirm it is Ready:
nemoclaw onboard --fresh --name {sandbox} --non-interactive --yes
nemoclaw {sandbox} status
2. Record its container id:
docker ps --filter label=openshell.ai/sandbox-name={sandbox} --format '{{.ID}}'
3. Control, to show the supported lifecycle still works:
nemoclaw {sandbox} stop
nemoclaw {sandbox} start
nemoclaw {sandbox} status
4. Now interrupt at the container level and clear the interruption:
docker stop {container id}
docker start {container id}
5. Run the documented recovery command:
nemoclaw {sandbox} start
6. Run the other recovery command:
nemoclaw {sandbox} recover
7. Check the state, and confirm the container is healthy:
nemoclaw {sandbox} status
docker ps --filter label=openshell.ai/sandbox-name={sandbox}

Second path, from a daemon restart:
8. On a Ready sandbox, run: sudo systemctl restart docker
9. Wait for the daemon, then run: nemoclaw {sandbox} start
```

## Expected Result

Step 5: `start` waits for the managed gateway supervisor within its bounded wait,
completes startup recovery, and exits 0; status then reports Ready.

Step 6: `recover` exits 0 on a sandbox whose container is running and healthy, or
fails with a diagnostic that names what actually failed and what to do next.

Step 9: `start` either completes recovery and exits 0, or exits non-zero with
actionable retry guidance. A sandbox must not become unrecoverable because the
container runtime restarted.

## Actual Result

```text
Control (step 3), on the same sandbox and build - works correctly:

nemoclaw {sandbox} stop exit 0 phase Stopped
nemoclaw {sandbox} start exit 0 37s, phase Ready, container Up (healthy)
connect --probe-only exit 0 result=ready firstFailedObservation=none

Case (steps 4-7) - wedged:

docker stop / docker start exit 0 container Up (healthy)
nemoclaw {sandbox} start exit 1 60s of "Status: Error" polls
nemoclaw {sandbox} recover exit 1 phase still Error
nemoclaw {sandbox} status Phase: Error
docker ps container Up (healthy)

`start` output:

Sandbox '{sandbox}' is already running.
Restoring sandbox startup state...
Waiting for sandbox '{sandbox}' to be ready...
Status: Error (3s elapsed) ... Status: Error (57s elapsed)
Sandbox '{sandbox}' entered 'Error' state.
Run: nemoclaw {sandbox} logs --follow
Run: nemoclaw {sandbox} status

`recover` output. Both hosts report result=failed failedStage=gateway, but the
probe's own attribution fields disagree between them, and on the first host the
probe reports that nothing failed at all:

host 1: readiness.firstFailedObservation=none
readiness.firstDecision=missing
readiness.firstFallbackDecision=missing
result=failed failedStage=gateway total=184ms
host 2: readiness.firstFailedObservation=sandbox-identity
readiness.firstDecision=health
readiness.firstFallbackDecision=health
result=failed failedStage=gateway total=216ms

Sandbox '{sandbox}' is in 'Error' state.
Run: nemoclaw {sandbox} logs --follow
Run: nemoclaw {sandbox} status

Daemon-restart path (steps 8-9):

docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' no
after systemctl restart docker, container state exited (waited 120s)
nemoclaw {sandbox} start exit 1
OpenShell could not start sandbox '{sandbox}' (exit 1): code: 'The system is
not in a state required for the operation's execution', message: "sandbox must
be Stopped to start (current phase: Error)".

`connect --probe-only`, tried as a last resort, also exits 1
(result=failed failedStage=gateway) and leaves the phase at Error.

Repeating `docker restart` then `start` reproduces the identical wedge. No
command in the CLI moved the sandbox out of Error.

Confirmed independently on a second DGX Station host, same build, with the same
control-then-case sequence: control stop/start exit 0 and Ready; docker
stop/start then `start` exit 1 after 62s of Error polls; `recover` exit 1;
`connect --probe-only` exit 1; final phase Error with the container Up (healthy).
```

## Logs

```text
Sequence captured on one fresh sandbox, in order:

onboard exit 0 Phase: Ready
restart policy no
nemoclaw stop (control) exit 0 Phase: Stopped
nemoclaw start (control) exit 0 Phase: Ready 37s
docker stop exit 0
docker start exit 0 container Up (healthy)
nemoclaw start exit 1 Phase: Error 60s
nemoclaw recover exit 1 Phase: Error
systemctl restart docker exit 0 container exited
nemoclaw start exit 1 "sandbox must be Stopped to start
(current phase: Error)"
docker restart + start exit 1 Phase: Error 60s

Second host, same build, same sequence:

nemoclaw stop (control) exit 0 Phase: Stopped
nemoclaw start (control) exit 0 Phase: Ready
docker stop / docker start exit 0 container Up (healthy)
nemoclaw start exit 1 Phase: Error 62s
nemoclaw recover exit 1 Phase: Error
connect --probe-only exit 1 result=failed failedStage=gateway
final Phase: Error, container Up (healthy)
```

## Related Bugs / not duplicate of

Checked the open and closed NemoClaw issue list. Distinct from:

- **#8112** — `nemoclaw start` after `stop` leaves the sandbox Unhealthy in Error phase.
That is the CLI stop/start path, it is closed as fixed, and a control run in this same
session confirms it is still fixed on this build. Only the container-level interruption
path fails here.
- **#11790** — `recover` cannot return a Stopped sandbox to Ready. That starts from a
Stopped sandbox and `recover` never returns before its 300s timeout; here the sandbox is
in Error, the container is Up and healthy, and `recover` returns in under a second.
- **#11879** — unpausing a sandbox container leaves it permanently in Error. Different
trigger (pause/unpause); this one is container stop/start and a daemon restart.
- **#11165** — `dcode rebuild --yes` cannot recover a sandbox stuck in Error with a live
but unresponsive container. Different command and different agent; the container here is
healthy, and the failing commands are `start` and `recover`.
- **#11715** — `start` misdiagnoses an unreachable Docker daemon as a removed container.
There the daemon is down; here the daemon and the container are both up.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the container stop/start and Docker daemon restart paths, then trace the `start`, `recover`, and `connect --probe-only` entry points when the sandbox is in Error. Compare these paths with the documented `stop`/`start` control flow and inspect the reported phase and container health. Done means an interrupted sandbox can return to Ready or produces actionable recovery guidance without becoming permanently unusable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.