NVIDIA / NVIDIA/NemoClaw

Complete reboot persistence by conservatively restoring sandboxes and host forwards

Open
#7,174 0 comments 0 reactions 0 assignees View on GitHub
area: architecture area: install area: networking area: sandbox needs: design platform: linux
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Problem Statement

NemoClaw does not provide a complete boot-persistence contract for an always-on headless deployment.

#6903 and draft PR #6905 cover installing and owning a persistent OpenShell gateway service. That is necessary but not sufficient: after the gateway returns, NemoClaw must decide which registered sandboxes should run, wait for authoritative sandbox and agent readiness, and restore recorded host forwards without restarting healthy components or creating a recovery loop.

Current `main` (`4deebcf9cfd0d54c9b9b5fb1907d153a462818f3`) has useful pieces but no end-to-end mode:

- `src/lib/onboard/docker-driver-gateway-service.ts` can use a trusted packaged user unit.
- Standalone gateway launch remains a detached process when no unit is available.
- `src/lib/actions/sandbox/start.ts` is a one-shot start/probe.
- `src/lib/actions/sandbox/forward-recovery.ts` can reconcile one exact-owner forward and fails closed on foreign port ownership.
- `docs/manage-sandboxes/run-sandboxes.mdx`, `docs/get-started/quickstart-hermes.mdx`, and `docs/reference/commands.mdx` require manual sandbox/forward recovery after reboot.

A naive `Restart=always` unit around `nemoclaw recover` is unsafe because repair currently spans gateway, sandbox, agent, and forward mutation. #7173 is therefore a prerequisite.

## Desired Behavior

Provide an opt-in Linux service mode, naming TBD, that converges after reboot without an interactive login:

- one declared gateway lifecycle authority starts;
- only explicitly boot-enabled sandboxes start;
- intentionally stopped and already-healthy sandboxes are left alone;
- OpenShell `Ready` and agent readiness are awaited with a bounded slow-boot budget;
- recorded forwards are restored only after readiness;
- exact target-owned forwards are reused and foreign listeners are never killed/adopted;
- ambiguous states fail visibly without restarting the host gateway;
- uninstall removes only NemoClaw-owned artifacts.

Example product shape:

```bash
nemoclaw service install
nemoclaw service status
nemoclaw service uninstall
```

Build two explicit layers:

1. Keep #6903 as gateway-service authority and prevent duplicate gateway owners.
2. Add a narrow per-sandbox boot reconciler whose only mutation authority is to start an explicitly enabled stopped sandbox, wait for readiness, and restore exact-owner forwards.

Do not implement this as an infinite loop around current `recover`. Prefer a dedicated reconciliation action plus either a low-frequency conservative keeper/timer or a supervised foreground-forward contract.

Persist an explicit desired-running/service-enabled bit. Do not infer operator intent from Docker restart policy. Define how manual `stop`, `start`, rebuild, and maintenance interact with that bit and lock concurrent mutation.

## Constraints and Non-goals

- Prefer user services; require explicit operator approval before enabling lingering and document its security implications.
- Use validated absolute executable paths and direct argv, not shell interpolation or inherited caller environments.
- Persist only non-secret identifiers/state; never provider credentials, bearer tokens, or arbitrary onboarding environment.
- Validate unit ownership/modes, reject unsafe symlinks, write atomically, and verify identity before uninstall.
- Never modify vendor units or compete with an external supervisor.
- No Kubernetes orchestration, macOS LaunchAgent in the first slice, automatic rebuild at boot, policy/credential repair, or public remote exposure.

## Implementation Idea

A user template unit could depend on Docker/network and the single gateway authority, then invoke a dedicated boot-reconcile action:

```text
gateway healthy -> start boot-enabled sandbox -> wait OpenShell Ready
-> wait agent health -> reconcile recorded forwards -> verify ownership/reachability
```

Use bounded retry/backoff and a start limit that leaves a failed unit visible rather than flapping. Maintenance/rebuild must suppress keeper mutations.

Test pure unit rendering/escaping, desired-state transitions, lifecycle idempotency, foreign port conflicts, slow boot, multi-sandbox/custom-port isolation, systemd adapter states, tamper-safe uninstall, and a real reboot E2E without login.

## Acceptance Criteria

- A registered Linux sandbox can opt into documented boot persistence.
- Gateway ownership remains unique and compatible with #6903/#6905.
- Reboot recovers the gateway, explicitly enabled sandbox, agent, and forwards without login.
- Slow Hermes boot does not cause restart loops.
- A healthy/ambiguous gateway is never restarted by sandbox supervision.
- Foreign listeners are never killed or adopted.
- Manual lifecycle and maintenance operations have defined locking/desired-state semantics.
- Units/metadata contain no raw secrets and uninstall removes only NemoClaw-owned artifacts.

## Category

Platform

## Related work

#6903 and draft PR #6905 own gateway service installation. PR #6842 and #6576 concern lifecycle authority. #6972 is a separate post-reboot volume issue. #7173 blocks safe supervision.

Contributor guide

Open the contributing guide

Research direction

Start with #7173, #6903, and the gateway authority code in src/lib/onboard/docker-driver-gateway-service.ts, then inspect src/lib/actions/sandbox/start.ts and forward-recovery.ts. Use the documented lifecycle, state, ownership, and reboot tests as the acceptance map; done means a bounded, opt-in Linux service restores only enabled sandboxes and exact-owner forwards without duplicate gateway control or unsafe uninstall.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
devops, infrastructure, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.