NVIDIA / NVIDIA/NemoClaw

OpenClaw snapshots can copy active SQLite databases without writer quiescence

Open
#10,827 1 comment 0 reactions 0 assignees View on GitHub
integration: openclaw needs: design
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation Summary

- OpenClaw snapshot directories are archived while the gateway remains running; the snapshot action takes a NemoClaw lifecycle lock but does not drain OpenClaw database writers.
- Directory capture accepts tar exit statuses that include "files changed during archive", which is not a coherent SQLite/WAL snapshot guarantee.
- The OpenClaw manifest captures `agents/` recursively, including per-agent SQLite databases, using raw directory copy rather than NemoClaw's implemented `sqlite_backup` strategy.
- `nemoclaw agent` starts a separate `openclaw agent` process against the shared state root and does not acquire the snapshot lifecycle lock, so it can overlap capture.
- The separate missing global `state/openclaw.sqlite` declaration is already tracked in #10694; this report focuses on writer quiescence and raw nested-database capture.

## Description

NemoClaw has a strong generic SQLite state-file implementation: online backup, busy timeout, `PRAGMA quick_check`, staged atomic restore, stale WAL/SHM removal, and a post-restore write transaction. The Hermes manifest uses it. The OpenClaw manifest does not use it for OpenClaw's SQLite databases.

For OpenClaw, snapshot creation archives manifest-declared directories while the gateway remains live. The `agents/` directory can contain `agents//agent/openclaw-agent.sqlite`, but it is captured recursively as ordinary files. The lifecycle lock serializes NemoClaw host mutations; it is not acquired by the running gateway or by `openclaw agent` processes, so it does not quiesce SQLite writers.

The archive implementation also accepts tar output when files changed during archive. That is reasonable for best-effort filesystem content, but not for a database whose committed generation may span the main database and WAL while they are copied at different instants.

Expected behavior: an OpenClaw snapshot either uses SQLite's online backup API for every global and per-agent database while the runtime is live, or drains all processes that can write the shared state root before raw directory capture. Snapshot success should require database integrity verification before publication and after restore.

Related: #10694 already identifies the missing global OpenClaw state declaration and requires WAL-consistent migration snapshots. This issue is the smaller runtime snapshot contract needed independently of the 2.0 cutover.

## Reproduction Steps

Source-level reproduction against `main` at `0bf091c81e56ed8e4569a7bf3c9a0249443fac3d`:

1. Inspect `agents/openclaw/manifest.yaml`: `agents` is a snapshot directory; no OpenClaw SQLite file uses `strategy: sqlite_backup`.
2. Inspect `src/lib/actions/sandbox/snapshot.ts`: snapshot creation requires a running sandbox and calls backup under a NemoClaw lifecycle lock without stopping the OpenClaw gateway.
3. Inspect `src/lib/state/sandbox.ts` directory backup: selected directories are streamed through tar, and output is accepted for tar statuses that can include files changing during archive.
4. Inspect `src/lib/actions/sandbox/agent/passthrough.ts`: `nemoclaw agent` starts `openclaw agent` through `openshell sandbox exec` against the same runtime environment and does not take the snapshot lifecycle lock.
5. Create an OpenClaw snapshot while a gateway/agent process is actively writing a per-agent database. The current contract can archive the database, WAL, and SHM at different transaction generations and still report directory capture success.

A deterministic regression can avoid timing dependence: hold an uncheckpointed WAL transaction/open writer, mutate between reads of the main DB and sidecars in the directory-copy harness, and assert the old implementation accepts a snapshot that fails `PRAGMA quick_check` or lacks committed data. The corrected implementation should use `sqlite_backup` or prove writer quiescence.

## Environment

- OS: source-level analysis; runtime-independent
- Hardware: not applicable
- Node.js: not applicable
- Docker: not applicable
- NemoClaw: `main` at `0bf091c81e56ed8e4569a7bf3c9a0249443fac3d`
- OpenClaw integration manifest: repository-pinned OpenClaw

## Debug Output

No secret-bearing runtime bundle is needed for this source-level reproduction. Relevant implemented paths:

- `agents/openclaw/manifest.yaml` snapshot declarations
- `src/lib/state/sandbox.ts` directory and SQLite backup strategies
- `src/lib/actions/sandbox/snapshot.ts` running-sandbox snapshot flow
- `src/lib/actions/sandbox/agent/passthrough.ts` separate agent process
- `src/lib/state/state-file-restore.ts` verified SQLite restore machinery already available

The repository's live OpenShell upgrade E2E already runs `PRAGMA integrity_check` against global and per-agent OpenClaw databases. The missing assertion is integrity and committed-data preservation across snapshot create/restore while writers are active.

## Checklist

- [x] I confirmed the unsafe path from the current source and provided a deterministic regression shape
- [x] I searched existing issues; #10694 covers the global-state/2.0 migration requirement but not this focused live-snapshot writer-exclusion defect

Contributor guide

Open the contributing guide

Research direction

Start with agents/openclaw/manifest.yaml and read the snapshot flow in src/lib/actions/sandbox/snapshot.ts and src/lib/state/sandbox.ts, then inspect the separate agent path in src/lib/actions/sandbox/agent/passthrough.ts. Run the existing OpenShell upgrade E2E and add a deterministic active-writer snapshot/restore regression. Done means snapshots preserve committed data and pass SQLite integrity checks before publication and after restore.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, typescript
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.