NVIDIA / NVIDIA/NemoClaw

[DGX Spark][Recovery] v0.0.103 cannot relaunch legacy supervisor after reboot with Shields UP

Open
#9,315 3 comments 0 reactions 0 assignees View on GitHub
area: sandbox needs: design platform: dgx-spark
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation Summary

- On an official NemoClaw `0.0.103` / OpenShell `0.0.85` Docker sandbox, a real DGX Spark host reboot leaves only PID 1 (`openshell-sandbox`) and the persisted legacy workload `sleep infinity`; `nemoclaw-start` and the OpenClaw gateway are absent.
- `nemoclaw sandbox status dabusclaw-main --json` exits 1 with `gatewayState=sandbox_recovery_failed`, while `doctor --json` still sees the registry sandbox as Ready and the inference route/provider backend as reachable.
- One controlled official `nemoclaw sandbox recover dabusclaw-main` attempt exits 1. The container identity, backup count, startup command, and two-process topology remain unchanged, so recovery fails closed before recreation.
- The installed v0.0.103 relaunch code requires a complete sandbox-user state backup before transactionally replacing a legacy keepalive container. With Shields UP, the manifest-listed `credentials` directory is intentionally `0710 root:sandbox` and cannot be archived by the sandbox user; an earlier official rebuild manifest also records `credentials` as a failed backup directory.
- The exact internal failure layer remains unknown because the public `recover` output reports only generic automatic recovery failure. The backup/permission conflict is a high-confidence inference from the installed tagged source, live permissions, archive probes, and prior rebuild manifest—not a directly emitted controller diagnostic.

## Description

This appears to be a residual regression in the trusted legacy-supervisor relaunch added by #6677 for #6635.

After a real host reboot, OpenShell correctly resumes the registered Docker sandbox and runs its persisted exact legacy command:

```text
/opt/openshell/bin/openshell-sandbox (PID 1, root)
└─ sleep infinity (sandbox user)
```

The managed `nemoclaw-start` supervisor and OpenClaw gateway are missing. The supported manual recovery command then fails without replacing or mutating the container:

```text
$ nemoclaw sandbox recover dabusclaw-main
Probe failed: OpenClaw gateway is not running in 'dabusclaw-main' and automatic recovery failed.
exit=1
```

Expected: for an exact legacy `OPENSHELL_SANDBOX_COMMAND=sleep infinity` Docker sandbox with a stably absent supervisor, `recover` should perform the documented registry-scoped transactional relaunch and restore the managed gateway topology, or return a precise safe diagnostic identifying the failed gate.

Actual: it returns the generic failure above. Before and after the attempt, the same container remains, no new backup is created, the persisted command remains `sleep infinity`, and the process tree remains PID 1 plus `sleep`.

This is distinct from the general automatic boot-persistence work in #7174: the supported **manual** `recover` path itself is failing. It is also distinct from the old rebuild issue #3113: current rebuild can auto-handle Shields, but the legacy supervisor relaunch path appears to snapshot state without an equivalent Shields-aware contract.

No permission workaround, manual in-container launch, registry/database edit, custom systemd unit, or Shields-down workaround was used.

## Reproduction Steps

1. Onboard or officially rebuild an OpenClaw sandbox with NemoClaw `0.0.103`, Docker driver, and Shields UP.
2. Confirm that the sandbox and OpenClaw gateway are healthy before reboot.
3. Reboot the DGX Spark host normally.
4. Run:

```bash
nemoclaw sandbox status dabusclaw-main --json
nemoclaw sandbox recover dabusclaw-main
nemoclaw sandbox status dabusclaw-main --json
nemoclaw sandbox doctor dabusclaw-main --json
```

5. Inspect the in-container process tree and persisted Docker startup command.
6. Observe that only `openshell-sandbox` plus `sleep infinity` remain and that `recover` exits 1 without transactionally replacing the container.

Additional read-only isolation of the suspected backup gate:

1. Read the installed OpenClaw state manifest used by `backupSandboxState`.
2. As the sandbox user, test archive readability for each declared state directory.
3. Observe that the existing `credentials` directory cannot be listed or archived while Shields is UP (`0710 root:sandbox`). Other present state directories in this sandbox are archive-readable.
4. Observe that the earlier official rebuild manifest lists `credentials` in `failedBackupDirs`.

## Environment

- Hardware: NVIDIA DGX Spark / GB10
- OS: Ubuntu 24.04.4 LTS, aarch64
- Kernel: `6.17.0-1029-nvidia`
- GPU: NVIDIA GB10, driver `580.173.02`
- Docker Engine: `29.2.1`
- NemoClaw: `0.0.103`
- OpenShell: `0.0.85`
- OpenClaw: `2026.7.1`
- Installed NemoClaw source: tag `v0.0.103`, commit `db31c286129e878c3356eed49f76ab259561e47e`, clean detached HEAD
- `nemoclaw update --check`: current and latest maintained are both `0.0.103`
- `nemoclaw upgrade-sandboxes --check`: all sandboxes up to date
- Docker restart policy: `unless-stopped`
- Persisted sandbox command: exact `OPENSHELL_SANDBOX_COMMAND=sleep infinity`
- Shields: UP (lockdown active)

The external local-vLLM provider remained running and reachable throughout. It was not restarted or modified.

## Debug Output

An official debug bundle was generated with `nemoclaw debug --sandbox dabusclaw-main --output ...`:

```text
filename: nemoclaw-debug-dabusclaw-main-20260817.tar.gz
size: 19157 bytes
sha256: 1d6d942956a3d17346c7ae70c97806e60c234dbf4ee30d90398cf7d21c3a9e37
```

The full tarball is intentionally not attached to this public issue because host/network diagnostics can contain environment identifiers. It can be provided to an NVIDIA maintainer through a private channel after review.

Sanitized status/debug evidence:

```text
nemoclaw sandbox status ... --json
exit=1
found=true
phase=null
gatewayState=sandbox_recovery_failed
liveRoute=null
routeDrift=null
dockerPaused=false

nemoclaw sandbox doctor ... --json
exit=0
OpenShell connected
sandbox registry phase Ready
inference route reachable
provider backend reachable
OpenClaw 2026.7.1
Shields UP lockdown active
```

## Logs

```shell
# OpenShell gateway journal after boot
compute driver connected
resumed sandbox during gateway startup: phase=Ready
sandbox supervisor session accepted
OCSF PROC:LAUNCH sleep(...)

# Process tree after boot and after one official recover attempt
root 1 /opt/openshell/bin/openshell-sandbox
sandbox 123 sleep infinity

# Recovery result
gateway is not running and automatic recovery failed
exit=1

# Relevant live permissions while Shields is UP
0755 root:root /sandbox/.openclaw
0710 root:sandbox /sandbox/.openclaw/credentials
0600 root:sandbox /sandbox/.openclaw/devices/paired.json

# Sandbox-user checks
credentials: list/read/archive denied
other present manifest state directories: archive-readable
```

The `paired.json` readability warning appears separate: `devices` is runtime auth state excluded from snapshots, and pre-reboot gateway logs show successful inference responses until the reboot. It may be another ownership/readability issue, but it does not explain the suspected `credentials` snapshot gate.

## Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start with the installed v0.0.103 source around backupSandboxState and the manual nemoclaw sandbox recover entry point, then reproduce with Shields UP using the commands in the issue. Trace the failed backup or recovery gate and verify that an exact legacy sleep infinity sandbox either relaunches transactionally with the managed gateway restored or reports a precise safe diagnostic.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.