[Ubuntu 26.04][Sandbox] unpausing a sandbox container leaves it permanently in Error and the probe reports failure without naming any failed observation
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
Pausing a sandbox's container and then unpausing it leaves the sandbox permanently in the Error phase. The container returns to running, but NemoClaw never re-observes it: every later `connect --probe-only` exits non-zero, and `start` cannot clear the state either.
The probe's own diagnostics stop being useful at exactly the moment they are needed. While the container is paused the probe behaves correctly and names the first failed observation. After the unpause the probe still reports failure, but with `readiness.firstFailedObservation=none` and every per-observation timing and attempt count at zero, so the receipt says "failed" without identifying anything that failed.
This matters because pause/unpause is the documented way to exercise the probe's error path, and because the end state is unrecoverable through the commands the product offers for it.
- **Platform scope:** Reproduced on Ubuntu 26.04 x86_64 only; other platforms not tested this pass.
- **Regression:** No — a control on the same host and build shows the neighbouring lifecycle path is healthy. On a fresh sandbox, stop then start completes in 37s and returns Ready with a passing probe.
- **OpenShell issue:** No — the container is running after the unpause; it is NemoClaw's own readiness observation that never recovers.
## Environment
```text
Device: x86_64 server (no GPU)
OS: Ubuntu 26.04 LTS
Architecture: x86_64
Kernel: 7.0.0-22-generic
Docker: Docker version 29.6.1, build 8900f1d
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.125
OpenClaw: 2026.7.1
```
## Steps to Reproduce
1. Onboard an OpenClaw sandbox and confirm it is Ready.
2. Confirm the healthy baseline: `nemoclaw {sandbox} connect --probe-only`
3. Pause the sandbox's container:
```bash
CID="$(docker ps --filter label=openshell.ai/sandbox-name={sandbox} --format '{{.ID}}' | head -n 1)"
docker pause "$CID"
```
4. Probe while paused: `nemoclaw {sandbox} connect --probe-only`
5. Unpause and confirm the runtime shows it running:
```bash
docker unpause "$CID"
docker ps --filter id="$CID"
```
6. Probe again, several times, with time in between.
7. Try to clear the state: `nemoclaw {sandbox} status`, then `nemoclaw {sandbox} start`.
## Expected Result
After the unpause the sandbox returns to Ready and the probe exits 0 with a proved successful outcome, not retaining the earlier failed result. If a probe does report a failure, its receipt identifies which observation failed.
## Actual Result
Reproduced on two independent sandboxes on the same host.
```text
baseline probe exit 0 result=ready firstFailedObservation=none
probe while paused exit 1 result=failed firstFailedObservation=sandbox-identity
(second sandbox: gateway-health)
docker unpause exit 0 container reports running
probe after unpause x3 exit 1 result=failed firstFailedObservation=none
status Phase: Error
start exit 1 Phase: Error
```
After the unpause the container is up for minutes while the sandbox phase stays Error indefinitely. Three consecutive probes spaced 20s apart, and a further probe minutes later, all return the same result.
The post-unpause receipts carry no diagnostic content at all — every observation timing and attempt count is zero:
```text
readiness.sandbox-identity=0ms readiness.sandbox-identity.attempts=0
readiness.policy-get=0ms readiness.policy-get.attempts=0
readiness.gateway-health=0ms readiness.gateway-health.attempts=0
readiness.forward-health=0ms readiness.forward-health.attempts=0
readiness.inference-route=0ms readiness.inference-route.attempts=0
readiness.firstFailedObservation=none
lifecycleAction=skipped forwardAction=skipped result=failed failedStage=gateway
```
Control on the same host and build, on a separate fresh sandbox: stop then start exits 0, takes 37s, returns Phase Ready with the container healthy, and the following probe exits 0 with `result=ready`.
## Logs
```text
Gateway log at the moment the phase changes:
Sandbox failed to become ready sandbox_name={sandbox} reason=ContainerPaused Container is paused
That reason is recorded while the container is paused, which is correct. After the unpause
no further readiness transition is recorded and the phase is never revised.
Runtime view after the unpause:
docker ps reports Up 7 minutes (unhealthy)
Probe while paused (useful diagnostic, shown for contrast):
readiness.sandbox-identity=22ms readiness.sandbox-identity.attempts=1
readiness.firstFailedObservation=sandbox-identity
lifecycleAction=skipped forwardAction=skipped result=failed failedStage=gateway
```
## Related Issues / not duplicate of
- **#8112** — start after stop left the sandbox Unhealthy in Error and only rebuild recovered it. Closed as completed. Same end state and same platform family, but a different trigger, and it has **not** regressed: the control above ran that exact stop/start sequence on this build and it succeeded.
- **#10638** — start reaches Ready then fails launch-readiness epoch revalidation. Different stage and outcome; here the sandbox never reaches Ready again at all.
- **#9280** — `connect --probe-only` exits 1 after restoring the dashboard forward. That failure is about the forward stage; here the probe never reaches any stage and reports no failed observation.
Contributor guide
Research direction
Start by reproducing the pause/unpause sequence with Docker, then compare `connect --probe-only`, `status`, and `start` before and after unpausing. Use the gateway log and post-unpause readiness receipt as the diagnostic checkpoints. Done means the sandbox returns to Ready, the probe exits 0, and any failed probe identifies its first failed observation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100