NemoClaw cold-boot gateway recovery starts a gateway, then rolls it back

### Investigation Summary

- **Confirmed:** deterministic on this host — 4/4 cold boots show the identical
start-then-rollback sequence, on both a v0.0.83-era sandbox (after upgrade to
v0.0.90 + rebuild) and a sandbox freshly onboarded on v0.0.90 the same day,
ruling out sandbox age/legacy config.
- **Confirmed:** the gateway binary is not the problem — direct launch via
`nemoclaw-start` succeeds in ~0.5 s, 4/4; the recovery-started instance dies
before its first log line, consistent with `relaunch.finalize(false)` rolling
back the container after the managed supervisor probe fails
(`process-recovery.ts`).
- **Confirmed:** recovery's host-side checks invoke `sudo /usr/bin/lsof -i
:8080` (seen in auth log), which fails silently on hosts without passwordless
sudo; separately, `policy-add` without `--yes` crashes with a readline EOF on
non-tty stdin, and destroy + re-onboard silently drops user-added policy
presets and provider attachment.
- **Unknown:** why the managed supervisor probe fails on this host — whether
the probe's PID-1 identity/readiness contract is unsatisfiable in this
environment (hardened host, no passwordless sudo, default-deny sandbox
egress) or the probe races the gateway's startup.
- **Unknown:** whether rollback-on-probe-failure is ever correct when the
started gateway is healthy — i.e., whether the fix is the probe, the
rollback, or both.

### Description

After a host reboot, `nemoclaw recover` (scripted from a boot unit, per
the docs' "idempotent and safe to script" guidance) reproducibly fails to
restore the in-sandbox OpenClaw gateway. Trusted container recovery recreates
the sandbox container with its managed startup command, starts a gateway
process, then declares it unresponsive and rolls the container back — killing
the process it just started. The rolled-back gateway dies before writing a
single line to `/tmp/gateway.log`, and the printed remediation (`rebuild
--yes`, then destroy + re-onboard) does not change the outcome: both were
performed the same day and the next cold boot failed identically. The gateway
itself is healthy — launched directly it reaches HTTP-listening in ~0.5 s and
is fully operational, every boot. The transient recovery-started process is
observable alive for a short window before rollback, which can defeat
operators' process-existence health checks.

**`sudo lsof` assumption**: during recovery the host-side checks invoke
`sudo /usr/bin/lsof -i :8080 -sTCP:LISTEN -P -n` (observed via auth log,
`PWD=~/.nemoclaw/source`). On hosts without passwordless sudo this prompts/
fails silently inside a non-interactive flow. Port checks shouldn't need
root (`ss -tlnp` on own processes, or an HTTP probe).

**Where the defect appears to live (from reading v0.0.90 source)**
`src/lib/actions/sandbox/process-recovery.ts`: after a managed relaunch,
`waitForRecoveredSandboxGateway(..., requireManagedProbe: true)` gates success on
`confirmRecoveredSandboxGatewayManaged` (a pinned supervisor-action probe). When
that probe fails, `relaunch.finalize(false)` **rolls back the container,
destroying the just-started gateway** — producing a transient process that
external tooling can observe alive for a window before it vanishes (this
zombie defeated our own process-existence health checks until we added a
stability window). The probe failure itself, not gateway health, is the root
cause on this host; the gateway never lives long enough to log.

**Potential workaround (what we are currently doing, shared for other operators)**
Boot oneshot: bounded loop of `recover` → backgrounded `connect` → if the host
proxy is up but no in-sandbox gateway process exists, a supervised
`exec ... nohup /usr/local/bin/nemoclaw-start &` launch; success requires BOTH
planes stable across three probes / 30 s (defeats the rollback-zombie window).
This has restored the plane on 4/4 cold boots. We would happily delete this if the
release for this is fixed.

### Reproduction Steps

(Reproduced 4 times in a row on the same workstation)
1. Healthy sandbox, gateway up, all NemoClaw health probes green.
2. `sudo reboot`.
3. Container returns via docker restart policy (`unless-stopped`), but no
supervisor/gateway runs inside it (known behavior, #311/#2042).
4. Boot script calls `nemoclaw recover` → host-plane transport error
(gateway RPC not up yet) → `nemoclaw connect` (backgrounded) brings
the host Docker-driver gateway up, then runs in-container recovery.
5. **Failure**: trusted container recovery recreates the container with its
managed startup command, starts a gateway process, then declares it
unresponsive and rolls back — killing the process it just started:
```
OpenClaw gateway is not running inside the sandbox (sandbox likely restarted).
Recovering...
Recreating the sandbox container with its managed startup command...
Gateway process started but is not responding.
Check /tmp/gateway.log inside the sandbox for details.
The in-sandbox supervisor is not running, and trusted container recovery
could not restore a managed supervisor and healthy gateway.
Recreate the sandbox runtime to restore it:
nemoclaw 'factory-gw' rebuild --yes
If rebuild is blocked, destroy and re-onboard the sandbox to restore it.
```

6. `/tmp/gateway.log` inside the sandbox contains **nothing** from this attempt
— the recovery-started gateway dies before its first log line.

### Environment

- NemoClaw v0.0.90 (upgraded from v0.0.83 same day; bug identical on both),
OpenClaw 2026.6.10 (NemoClaw-pinned), OpenShell docker driver
- Ubuntu 24.04, kernel 6.8.0-136-generic, Docker 29.1.3
(build 29.1.3-0ubuntu3~24.04.2) with buildx, NVIDIA CDI
- Node v22.23.1 (host) / v22.22.2 (in-sandbox)
- Host hardened: no passwordless sudo for the operating user (relevant — see
side finding 1)
- Sandbox: `factory-gw`, Ollama local inference (qwen3:8b), Slack channel
(Socket Mode), one custom agent; egress policy default-deny with presets
- Boot integration: a systemd oneshot (runs as the operating user) that calls
`nemoclaw factory-gw recover` per the docs' "idempotent and safe to script"
guidance, with a bounded retry loop

### Debug Output

═══ System ═══

Wed Jul 22 09:58:52 PM CDT 2026
Linux clawbox 6.8.0-136-generic #136-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul 1 21:53:05 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
21:58:52 up 44 min, 1 user, load average: 0.33, 0.41, 0.43
total used free shared buff/cache available
Mem: 63168 2587 53765 33 7493 60581
Swap: 89519 0 89519

═══ Processes ═══

PID PPID CMD %MEM %CPU
19055 19043 openclaw 0.1 156
19036 19035 node /home/claw/.nvm/versio 0.1 97.1
19043 7985 openclaw 0.1 21.4
18968 1 /usr/lib/systemd/systemd -- 0.0 7.3
1265 1 /usr/bin/dockerd -H fd:// - 0.1 0.8
990 1 /usr/bin/containerd 0.1 0.8
7955 7450 openclaw 0.6 0.8
1890 1267 /usr/local/lib/ollama/llama 1.1 0.1
6289 6264 /opt/openshell/bin/openshel 0.0 0.0
3323 1 openshell-gateway[nemoclaw= 0.0 0.0
1 0 /sbin/init 0.0 0.0
17 2 [rcu_preempt] 0.0 0.0
360 2 [jbd2/nvme0n1p2-8] 0.0 0.0
41 2 [migration/4] 0.0 0.0
432 1 /usr/lib/systemd/systemd-jo 0.0 0.0
35 2 [migration/3] 0.0 0.0
1267 1 /usr/local/bin/ollama serve 0.0 0.0
6264 1 /usr/bin/containerd-shim-ru 0.0 0.0
47 2 [migration/5] 0.0 0.0
77 2 [migration/10] 0.0 0.0
83 2 [migration/11] 0.0 0.0
53 2 [migration/6] 0.0 0.0
71 2 [migration/9] 0.0 0.0
23 2 [migration/1] 0.0 0.0
29 2 [migration/2] 0.0 0.0
59 2 [migration/7] 0.0 0.0
65 2 [migration/8] 0.0 0.0

═══ GPU ═══

Wed Jul 22 21:58:52 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4070 ... Off | 00000000:01:00.0 Off | N/A |
| 0% 36C P8 3W / 220W | 5543MiB / 12282MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1890 C ...local/lib/ollama/llama-server 5534MiB |
+-----------------------------------------------------------------------------------------+

═══ Docker ═══

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b85abd9a2f5 nemoclaw-sandbox-local:factory-gw-1784770858826 "/opt/openshell/bin/…" About an hour ago Up 42 minutes (healthy) openshell-factory-gw-22acd851-acc7-41e5-a11c-d15c709e467c
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
9b85abd9a2f5 openshell-factory-gw-22acd851-acc7-41e5-a11c-d15c709e467c 136.75% 657.4MiB / 61.69GiB 1.04% 3.08MB / 2.18MB 23.4MB / 13.9MB 60

═══ OpenShell ═══

(openshell not found, skipping)
(openshell not found, skipping)
(openshell not found, skipping)
(openshell not found, skipping)

═══ Onboard Session ═══

{
"version": 1,
"sessionId": "1784770864411-8daf11f0-d2f0-4345-a938-b5de2c2b23e9",
"status": "complete",
"resumable": false,
"mode": "non-interactive",
"startedAt": "2026-07-23T01:41:04.411Z",
"updatedAt": "2026-07-23T02:09:43.334Z",
"sandboxName": "factory-gw",
"provider": "ollama-local",
"model": "qwen3:8b",
"endpointUrl": null,
"credentialEnv": null,
"hermesAuthMethod": null,
"preferredInferenceApi": "openai-completions",
"compatibleEndpointReasoning": null,
"nimContainer": null,
"toolDisclosure": "progressive",
"observabilityEnabled": false,
"observabilityRequestedExplicitly": false,
"hermesToolGateways": [],
"policyPresets": [
"npm",
"pypi",
"huggingface",
"brew",
"local-inference",
"openclaw-pricing",
"slack",
"claude-code",
"github"
],
"gpuPassthrough": true,
"lastStepStarted": "policies",
"lastCompletedStep": "policies",
"failure": null,
"machine": {
"version": 1,
"state": "complete",
"stateEnteredAt": "2026-07-23T01:42:17.878Z",
"revision": 10
},
"steps": {
"preflight": {
"status": "complete",
"startedAt": null,
"completedAt": "2026-07-23T01:41:04.411Z",
"error": null
},
"gateway": {
"status": "complete",
"startedAt": null,
"completedAt": "2026-07-23T01:41:05.534Z",
"error": null
},
"sandbox": {
"status": "complete",
"startedAt": "2026-07-23T01:41:06.304Z",
"completedAt": "2026-07-23T01:41:45.946Z",
"error": null
},
"provider_selection": {
"status": "complete",
"startedAt": null,
"completedAt": "2026-07-23T01:41:05.572Z",
"error": null
},
"inference": {
"status": "complete",
"startedAt": null,
"completedAt": "2026-07-23T01:41:05.629Z",
"error": null
},
"openclaw": {
"status": "complete",
"startedAt": null,
"completedAt": "2026-07-23T01:41:46.733Z",
"error": null
},
"agent_setup": {
"status": "skipped",
"startedAt": null,
"completedAt": null,
"error": null
},
"policies": {
"status": "complete",
"startedAt": "2026-07-23T01:41:46.764Z",
"completedAt": "2026-07-23T01:41:52.746Z",
"error": null
}
}
}

═══ Kernel Messages ═══

(kernel messages skipped: dmesg access is restricted for this user; /proc/sys/kernel/dmesg_restrict=1 prevents non-root users from reading kernel logs.
Re-run with `sudo nemoclaw debug --quick` to include kernel logs in this report.
Note: privileged diagnostics and kernel logs may contain sensitive data; review before sharing.)

[debug] Done. If filing a bug, run with --output and attach the tarball to your issue:
[debug] nemoclaw debug --output /tmp/nemoclaw-debug.tar.gz

[operator note: two process-list lines removed (unrelated host services: VPN daemon, storage server) — no other edits]

### Logs

```shell
Full nemoclaw debug tarball generated but withheld — it includes complete host network diagnostics (ip-addr/ss/lsof/routes) not relevant to this failure. Targeted logs (boot-recover log, systemd unit journal, in-sandbox gateway.log for both failing and direct-launch cases) available on request.
```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Start in src/lib/actions/sandbox/process-recovery.ts and trace waitForRecoveredSandboxGateway, confirmRecoveredSandboxGatewayManaged, and relaunch.finalize(false). Reproduce nemoclaw recover after a cold boot, then compare the managed supervisor probe with the sudo lsof host check and the gateway logs. Done means recovery retains a healthy, HTTP-listening gateway without rolling back the container, with the failure mode covered by regression checks.

Written by the indexing model from the issue text.