Complete reboot persistence by conservatively restoring sandboxes and host forwards
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem Statement
NemoClaw does not provide a complete boot-persistence contract for an always-on headless deployment.
#6903 and draft PR #6905 cover installing and owning a persistent OpenShell gateway service. That is necessary but not sufficient: after the gateway returns, NemoClaw must decide which registered sandboxes should run, wait for authoritative sandbox and agent readiness, and restore recorded host forwards without restarting healthy components or creating a recovery loop.
Current `main` (`4deebcf9cfd0d54c9b9b5fb1907d153a462818f3`) has useful pieces but no end-to-end mode:
- `src/lib/onboard/docker-driver-gateway-service.ts` can use a trusted packaged user unit.
- Standalone gateway launch remains a detached process when no unit is available.
- `src/lib/actions/sandbox/start.ts` is a one-shot start/probe.
- `src/lib/actions/sandbox/forward-recovery.ts` can reconcile one exact-owner forward and fails closed on foreign port ownership.
- `docs/manage-sandboxes/run-sandboxes.mdx`, `docs/get-started/quickstart-hermes.mdx`, and `docs/reference/commands.mdx` require manual sandbox/forward recovery after reboot.
A naive `Restart=always` unit around `nemoclaw recover` is unsafe because repair currently spans gateway, sandbox, agent, and forward mutation. #7173 is therefore a prerequisite.
## Desired Behavior
Provide an opt-in Linux service mode, naming TBD, that converges after reboot without an interactive login:
- one declared gateway lifecycle authority starts;
- only explicitly boot-enabled sandboxes start;
- intentionally stopped and already-healthy sandboxes are left alone;
- OpenShell `Ready` and agent readiness are awaited with a bounded slow-boot budget;
- recorded forwards are restored only after readiness;
- exact target-owned forwards are reused and foreign listeners are never killed/adopted;
- ambiguous states fail visibly without restarting the host gateway;
- uninstall removes only NemoClaw-owned artifacts.
Example product shape:
```bash
nemoclaw service install
nemoclaw service status
nemoclaw service uninstall
```
Build two explicit layers:
1. Keep #6903 as gateway-service authority and prevent duplicate gateway owners.
2. Add a narrow per-sandbox boot reconciler whose only mutation authority is to start an explicitly enabled stopped sandbox, wait for readiness, and restore exact-owner forwards.
Do not implement this as an infinite loop around current `recover`. Prefer a dedicated reconciliation action plus either a low-frequency conservative keeper/timer or a supervised foreground-forward contract.
Persist an explicit desired-running/service-enabled bit. Do not infer operator intent from Docker restart policy. Define how manual `stop`, `start`, rebuild, and maintenance interact with that bit and lock concurrent mutation.
## Constraints and Non-goals
- Prefer user services; require explicit operator approval before enabling lingering and document its security implications.
- Use validated absolute executable paths and direct argv, not shell interpolation or inherited caller environments.
- Persist only non-secret identifiers/state; never provider credentials, bearer tokens, or arbitrary onboarding environment.
- Validate unit ownership/modes, reject unsafe symlinks, write atomically, and verify identity before uninstall.
- Never modify vendor units or compete with an external supervisor.
- No Kubernetes orchestration, macOS LaunchAgent in the first slice, automatic rebuild at boot, policy/credential repair, or public remote exposure.
## Implementation Idea
A user template unit could depend on Docker/network and the single gateway authority, then invoke a dedicated boot-reconcile action:
```text
gateway healthy -> start boot-enabled sandbox -> wait OpenShell Ready
-> wait agent health -> reconcile recorded forwards -> verify ownership/reachability
```
Use bounded retry/backoff and a start limit that leaves a failed unit visible rather than flapping. Maintenance/rebuild must suppress keeper mutations.
Test pure unit rendering/escaping, desired-state transitions, lifecycle idempotency, foreign port conflicts, slow boot, multi-sandbox/custom-port isolation, systemd adapter states, tamper-safe uninstall, and a real reboot E2E without login.
## Acceptance Criteria
- A registered Linux sandbox can opt into documented boot persistence.
- Gateway ownership remains unique and compatible with #6903/#6905.
- Reboot recovers the gateway, explicitly enabled sandbox, agent, and forwards without login.
- Slow Hermes boot does not cause restart loops.
- A healthy/ambiguous gateway is never restarted by sandbox supervision.
- Foreign listeners are never killed or adopted.
- Manual lifecycle and maintenance operations have defined locking/desired-state semantics.
- Units/metadata contain no raw secrets and uninstall removes only NemoClaw-owned artifacts.
## Category
Platform
## Related work
#6903 and draft PR #6905 own gateway service installation. PR #6842 and #6576 concern lifecycle authority. #6972 is a separate post-reboot volume issue. #7173 blocks safe supervision.
Contributor guide
Research direction
Start with #7173, #6903, and the gateway authority code in src/lib/onboard/docker-driver-gateway-service.ts, then inspect src/lib/actions/sandbox/start.ts and forward-recovery.ts. Use the documented lifecycle, state, ownership, and reboot tests as the acceptance map; done means a bounded, opt-in Linux service restores only enabled sandboxes and exact-owner forwards without duplicate gateway control or unsafe uninstall.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- devops, infrastructure, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100