NVIDIA / NVIDIA/NemoClaw

[DeepSeek Harness 4/8] Preserve and recover accepted DeepSeek Harness state

Open
#9,332 1 comment 0 reactions 0 assignees View on GitHub
area: integrations area: onboarding area: sandbox integration: deepseek-harness
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

Preserve, rebuild, restore, and recover only the DeepSeek Harness state accepted by Gate 1 through NemoClaw's shared lifecycle contracts.

Parent epic: #9328

Depends on:

- #9329, **[DeepSeek Harness 1/8] Accept product scope, ownership, threat model, and compatibility matrix**;
- #9330, **[DeepSeek Harness 2/8] Produce a reproducible DeepSeek Harness candidate runtime**;
- #9331, **[DeepSeek Harness 3/8] Let users onboard and use the accepted Web and headless surfaces**.

Gate 5 supplies reconstructed managed-inference configuration. Gate 6 supplies the final trust and policy boundary. Gate 7 owns exact live release qualification. Gate 8 alone owns public activation.

## Problem Statement

DeepSeek Harness writes several kinds of state beneath `$DSH_HOME`, may use a separate `$DSH_AGENTS_HOME`, and treats the invoking directory as the workspace.

Known paths include session logs, content-addressed attachments, Web domain storage, settings, credentials, profiles, user patches, anonymous identity, generated package symlinks, and caches. These paths do not share one portability or trust contract.

Blindly snapshotting the entire home could:

- preserve an upstream provider credential or project/home `.env` secret;
- allow restored settings or a Cordis patch to replace managed inference or policy;
- reactivate an unreviewed plugin, profile, skill, or executable project-trust decision;
- preserve machine-local package symlinks and generated `node_modules`;
- race a running Web/headless process and capture inconsistent JSON or append-only state;
- claim migration between prerelease formats that upstream does not support.

## Desired Behavior

The candidate manifest and shared NemoClaw state engine derive one exact state plan from the Gate 1 classification.

Restart, same-version rebuild, snapshot, restore, recovery, clone where supported, and destroy preserve only accepted user data. Managed profiles, route and policy overlays, package trees, caches, indexes, and runtime identity are reconstructed from trusted desired state.

Credential, telemetry-identity, direct-provider, and executable-trust state never enters a portable snapshot. A restore cannot outrank the root-owned managed overlay.

A later DSH version is accepted only after explicit format-compatibility evidence. Otherwise upgrade fails before destructive mutation and retains a tested rollback path.

## Required state inventory

Implementation must produce a repository-owned state contract covering every observed write from a bounded Web and headless run.

At minimum, classify:

| Path or state class | Proposed v1 treatment |
| --- | --- |
| `$DSH_HOME/sessions/` | Confidential persistent state; preserve only complete accepted session records. |
| `$DSH_HOME/attachments/` | Confidential content-addressed persistent state when accepted; preserve objects and references consistently. |
| `$DSH_HOME/storages/` | Persistent only when the accepted Web workspace/domain records require it. |
| Invoking workspace | Preserve through the shared NemoClaw workspace contract, not by duplicating it inside DSH-home backup. |
| `settings.yaml` | Allowlist-merge only accepted user preferences; reconstruct provider, model-route, permission, preset, and policy values. |
| `profiles/*/package.json` and `cordis.patch.yml` | Reconstruct shipped/managed profiles; preserve no executable user profile unless Gate 1 explicitly accepts and sanitizes it. |
| `$DSH_HOME/cordis.patch.yml` | Prohibited from portable restore unless Gate 1 defines a narrow non-security allowlist; it must never outrank managed authority. |
| `$DSH_AGENTS_HOME` and skill roots | Reconstruct bundled resources; user/project skills and executable trust follow the Gate 1 policy and do not become trusted through restore. |
| `.credentials.yaml`, `$DSH_HOME/.env`, project `.env` | Prohibited credential state. Never snapshot or restore. |
| `.anonymous-user-id` | Ephemeral/prohibited telemetry identity; do not restore. |
| Generated profile `node_modules` and package symlinks | Reconstructed from the exact image; do not snapshot or restore. |
| Caches, in-memory/search indexes, temporary files, logs | Reconstructed or ephemeral; do not restore unless Gate 1 explicitly records a bounded diagnostic exception. |
| Browser local storage/cookies/tokens | Outside server snapshot unless the accepted authentication component defines a separate revocable contract. |

If observed writes introduce another path, add it to the contract before closure. Unclassified paths fail the state-plan validation rather than being captured by a broad glob.

## Process and consistency contract

Follow the shared or isolated `$DSH_HOME` decision from Gate 1.

For shared state, snapshot and rebuild must use a proven quiescence or online-consistency protocol. Stopping only the Web process is insufficient if a headless process, worker, background job, or attachment write remains active.

For isolated homes, each surface receives an independent inventory and snapshot plan. Documentation must not claim cross-surface session continuity.

The state engine must preserve:

- session-to-attachment reference integrity;
- whole-file atomicity for accepted `storages/` records;
- owner-only modes and confidential Shields classification;
- symlink and traversal boundaries;
- the exact runtime/package/image identity associated with the snapshot;
- a clear incomplete/corrupt-state failure instead of silent partial restore.

## Lifecycle behavior

- **Restart:** reuse live accepted state without copying or reclassifying it; reconstruct final managed configuration before the surface becomes ready.
- **Rebuild:** capture accepted state, recreate from the exact managed image and desired startup profile, restore accepted state, reapply Shields, and verify runtime identity before readiness.
- **Snapshot/backup:** quiesce or prove consistent capture, produce an inventory and digest, sanitize prohibited material, and publish only after validation succeeds.
- **Restore:** verify provenance and format compatibility, restore to a staging location, validate structure and ownership, atomically publish accepted state, reconstruct managed files, and then start DSH.
- **Recovery:** distinguish process/forward failure from corrupt or incompatible state and give actionable commands without replacing accepted state silently.
- **Clone:** when the shared lifecycle supports clone, assign new sandbox/runtime/browser identities and preserve only portable content; do not clone credentials, telemetry identity, forwards, or machine-local package state.
- **Destroy:** stop all owned processes and forwards, remove sandbox-owned state through the selected compute-runtime provider, and clear registry metadata without touching unrelated host data.
- **Upgrade:** verify an explicit source-to-target state compatibility receipt before mutation. On missing or failed compatibility, retain the original runtime and state or restore the tested rollback snapshot.

## Scope

- Manifest-derived DSH state plan and repository-owned state-contract documentation.
- Persistent, allowlist-restored, reconstructed, prohibited, and ephemeral path rules.
- Confidentiality, ownership, file-mode, symlink, traversal, size, and bounded-inventory validation.
- Quiescence or online-consistency handling for all accepted Web/headless processes and workers.
- Shared restart, rebuild, backup, snapshot, restore, recovery, clone, upgrade, and destroy integration.
- Managed-overlay reconstruction after restore.
- Credential and executable-trust exclusion.
- Corrupt, incomplete, incompatible, and wrong-runtime snapshot errors.
- Candidate-only deterministic lifecycle tests and evidence.

## Constraints and Non-goals

- Do not snapshot `$DSH_HOME` wholesale.
- Do not use an unbounded glob to admit unknown future state.
- Do not capture or restore provider credentials, route credentials, `.env` values, `.credentials.yaml`, browser secrets, or `.anonymous-user-id`.
- Do not restore a model route, provider selection, permission mode, profile patch, external plugin, executable skill, or project trust decision as trusted desired state.
- Do not restore generated `node_modules`, package symlinks, caches, search indexes, or image-bundled profiles.
- Do not snapshot while an accepted writer remains active unless an online-consistency contract proves the result.
- Do not silently drop a missing attachment or publish a session that refers to an absent object.
- Do not promise migration between arbitrary DSH release candidates.
- Do not mutate the existing sandbox before source-to-target upgrade compatibility and rollback readiness are proven.
- Do not add a DeepSeek Harness-specific Docker, Podman, snapshot, recovery, clone, or destroy implementation.
- Do not activate public support in this issue.

## Observable acceptance tests

### Inventory and sanitization

- A bounded Web/headless workload records every runtime write; every path maps to one state-contract entry and an unknown path fails validation.
- Canary values placed in `.credentials.yaml`, home/project `.env`, direct-provider settings, `.anonymous-user-id`, logs, caches, profile plugins, and executable trust files never appear in a snapshot, manifest, diagnostic, or restore staging tree.
- Only the accepted user-preference keys survive an allowlist merge; managed model, provider, permission, preset, telemetry, and policy keys are reconstructed.
- Generated profile packages, symlinks, and caches come from the exact image after rebuild rather than the snapshot.
- Path traversal, external symlink, hard-link escape, unexpected file type, unsafe mode, over-size file, and unbounded-entry cases fail before publication.

### Same-version lifecycle

- Create a deterministic session, accepted attachment, workspace change, Web storage record when applicable, and allowlisted preference.
- Restart and prove all accepted content remains and the final managed overlay still wins.
- Snapshot, rebuild from the exact image digest, restore, and independently verify session history, attachment digest/reference, workspace content, storage record, preference, modes, and runtime identity.
- Interrupt capture and restore at each publication boundary; recovery selects either the complete old state or complete new state, never a mixed partial tree.
- A corrupt session record, missing attachment, malformed storage document, or incompatible state marker fails with an actionable error and preserves the original state.
- Destroy removes only the selected sandbox's owned state, processes, forwards, and registry row.
- Clone, when in accepted scope, produces a distinct sandbox identity and browser/auth state while preserving only portable content.

### Web/headless consistency

- Shared-home mode proves that every writer is quiesced or covered by the accepted consistency protocol before capture.
- A deliberately active headless turn, background job, attachment write, or Web storage update blocks or safely participates in snapshot according to the contract.
- Isolated-home mode captures each surface separately and never merges colliding sessions or settings implicitly.

### Upgrade refusal and rollback

- Same-version rebuild succeeds from the exact package and image identities.
- A target version without an accepted state-compatibility receipt fails before sandbox or state mutation.
- A target with an intentionally incompatible session/storage marker fails, retains the source snapshot, and restarts the original exact image.
- A compatibility-approved target must pass migration, post-restore invariant, and rollback tests before its receipt is accepted; this issue does not pre-approve a future version.

### Provider neutrality

- Deterministic tests inject at least two compute-runtime provider identities without changing the DSH state plan.
- Shared provider lifecycle functions receive manifest-derived state inputs and contain no `deepseek-harness` identity comparison.

## Required evidence

Attach or link a sanitized candidate lifecycle manifest containing:

- NemoClaw commit, Gate 1 decision revision, DSH package integrity, image digest, architecture, OpenShell version, compute runtime, and policy/configuration digests;
- the complete observed-write inventory and accepted state-plan digest;
- pre/post session IDs and log digests, attachment object/reference digests, storage-record digests, workspace oracle, and allowlisted preference result;
- snapshot, staging, atomic-publication, Shields, rebuild, restore, recovery, clone if applicable, destroy, and rollback results;
- negative canary, symlink/traversal, corruption, interruption, concurrency, and version-incompatibility results;
- proof that no credential, telemetry identity, executable trust state, or generated package tree entered the evidence.

## Documentation impact

- Add a repository-owned DeepSeek Harness state contract naming persistent, allowlist-restored, reconstructed, prohibited, and ephemeral paths.
- Document same-version restart, rebuild, snapshot, restore, recovery, clone if supported, destroy, and later-version refusal/rollback semantics for maintainers.
- Document shared versus isolated Web/headless state and snapshot quiescence.
- Add actionable recovery copy for corrupt, incomplete, incompatible, and prohibited state.
- Defer public operations documentation to Gate 8.

## Acceptance Criteria

- [ ] Gates 1 through 3 are complete and the implemented state plan matches their accepted surfaces and process model.
- [ ] Every observed DSH write is classified; unknown paths fail closed.
- [ ] Sessions, attachments, Web storage, workspace, settings, profiles, agents/skills, credentials, anonymous identity, packages, caches, logs, and browser state have explicit treatment.
- [ ] Snapshot and restore preserve only accepted state with correct ownership, modes, confidentiality, and reference integrity.
- [ ] Managed route, policy, preset, permission, and startup configuration reconstruct and outrank restored user state.
- [ ] Credentials, `.env` values, `.anonymous-user-id`, direct-provider settings, executable trust, generated package trees, and caches never enter portable state.
- [ ] Shared-home capture proves consistency across every writer, or isolated homes remain explicitly separate.
- [ ] Restart, same-version rebuild, snapshot, restore, recovery, destroy, and clone when accepted pass deterministic independent oracles.
- [ ] Interrupted, corrupt, incomplete, traversing, linked, oversized, and incompatible state fails safely without destroying the last good state.
- [ ] Later-version upgrade requires an explicit compatibility receipt and tested rollback; arbitrary RC migration remains unsupported.
- [ ] The lifecycle uses shared provider contracts without DeepSeek Harness-specific compute-runtime branches.
- [ ] Candidate lifecycle evidence and maintainer state/recovery documentation are complete and credential-free.
- [ ] Public selection and the active shipped-image cohort remain unchanged.

## Category

Feature

## Checklist

- [x] I searched existing issues and this is not a duplicate.
- [x] I described the problem and desired behavior.

Contributor guide

Open the contributing guide

Research direction

Start with the Gate 1 classification, the manifest-derived state plan, and the shared lifecycle functions; no implementation files are named in the issue. Run the candidate deterministic lifecycle and inventory tests described here, then verify that every observed path is classified, prohibited material is excluded, and restart, rebuild, snapshot, restore, recovery, and upgrade behavior meet the stated invariants.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.