NVIDIA / NVIDIA/NemoClaw

[Epic] Qualify Hermes as an experimental portable agent on rootless Podman

Open
#9,211 3 comments 0 reactions 1 assignee Claimed by @senthilr-nv View on GitHub
area: onboarding area: sandbox integration: hermes platform: container security
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Outcome

Hermes is an agent-specific experimental Portable capability on one defined Linux and rootless Podman matrix.

A qualifying run uses one durable current-user Podman authority and Hermes-specific startup, health, authentication, policy, state, recovery, and cleanup contracts. This Epic does not widen the OpenClaw Portable predicate or establish production Podman-provider support.

## Product decision

- This Epic owns experimental Hermes Portable qualification.
- OpenClaw Epic #9200 remains independent and may close without this Epic.
- #7744 remains the authority for production Podman-provider architecture and support.
- Hermes uses its own lifecycle strategy. Do not widen the OpenClaw predicate.
- Passing deterministic tests or one field run is evidence. It is not a support decision.
- Hermes must consume the shared Personal-profile policy owner and its mandatory `personal-open-internet` policy. Hermes must not duplicate the preset or policy composition.
- ACP package and import availability in a Hermes image does not establish ACP support. This Epic does not qualify ACP sessions, editor compatibility, workspace mapping, transport, or authorization.

## Accepted qualification matrix

Each implementation candidate and field result must bind to one commit SHA. Do not combine evidence from different revisions or matrices.

The accepted initial matrix is:

- Ubuntu 22.04 x86_64 with cgroups v2;
- CPU available through the current user's `app.slice` before NemoClaw starts;
- rootless Podman 5.7.0 through the current-user socket;
- Node.js 22.19.0 or newer within the repository's Node.js 22 contract;
- the OpenShell and Hermes versions pinned by the tested NemoClaw commit;
- local Ollama with `qwen3-vl:4b` for the first credential-free inference qualification;
- one NVIDIA L40S GPU-enabled run and one separate `--no-sandbox-gpu` run.

A Docker client or daemon may exist as unrelated host state. Qualification must:

- record the Docker socket authority and a credential-free inventory before NemoClaw changes the host;
- reject any preexisting NemoClaw-managed or OpenShell-managed Docker artifact;
- clear ambient Docker and Podman selectors before product commands;
- bind all Hermes receipts and product containers to the qualified current-user Podman socket;
- require Docker containers, images, volumes, and user-defined networks to remain unchanged;
- compare built-in Docker networks by stable semantics across a verified reboot;
- reject Docker fallback, a product-owned Docker artifact, authority ambiguity, or unexplained Docker mutation.

Do not stop, start, delete, reconfigure, or change permissions on an unrelated Docker service or socket.

## Accountable ownership

- Epic coordination, delivery order, and qualification evidence: `senthilr-nv`.
- Product scope and activation decisions: NemoClaw maintainers through recorded decisions on this Epic.
- #9203 owns the completed core lifecycle implementation.
- #9608 owns the completed schema-5 Hermes Portable uninstall contract, delivered through PR #9903.
- #9921 and merged PR #10006 own the original local `COPY` and remote `ADD` option correction lineage.
- #10007 and PR #10023 own the remaining classic-builder `RUN` compatibility, runtime authority, readiness settlement, UID-agnostic rootless regression, and protected staged-image build.
- Repository owners in `CODEOWNERS` retain review authority for lifecycle, agent, test, workflow, and documentation paths.
- #7744's owner must be consulted before a production-provider or broader support decision.

## Delivered foundation

- #9188 establishes fail-before-effects CPU admission through every required current-user cgroup boundary.
- #9189 supplies the general receipt-bound Portable retirement foundation.
- #9203 and PR #9424 implement schema-5 Hermes receipts, receipt-owned onboarding, start, stop, connect, probe, recovery, authenticated health, policy binding, and mutation fences.
- PRs #9571 and #9579 provide the production-shaped installer-to-active regression and private fixture custody.
- #9577 and PR #9578 establish network-before-alias ordering.
- #9587, PR #9641, and PR #9665 establish non-overlapping host-gateway authority and fail-closed retired-alias evidence.
- #9596 and PR #9632 start receipt-owned Ollama under the qualified current-user Podman authority.
- #9634's runtime behavior was delivered through PR #9635. Hermes Portable remains excluded from stock managed-runtime activation.
- PR #9694 pins the Portable rootless Podman runtime. PR #9714 establishes the Portable subnet and retired-subnet recovery. PR #9747 probes the sandbox's Hermes API port.
- PR #9790 aligns Hermes Portable with OpenShell 0.0.106.
- PRs #9867, #9881, #9906, and #9908 repair Podman proxy selection, acknowledged container identity, inference validation timing, and timed-out inspect handling.
- PRs #9912, #9915, #9922, and #9930 establish fail-closed sandbox absence detection, root-staged build-context custody, and removal of the local `COPY --chmod` incompatibility.
- PR #9837 binds the protected workflow to the reviewed Hermes base-image index.
- #9608 and PR #9903 implement schema-5 receipt-owned uninstall and retirement. PR #9870 separately makes Bedrock adapter shutdown fail closed before evidence retirement.
- PR #9940 pins the Portable launch Podman runtime. PRs #9951 and #9994 preserve receipt-owned Portable destroy identity across the shared cleanup boundary.
- PR #9964 adds bounded lifecycle recovery timing diagnostics. Timing data remains measurement, not an SLO.
- The replay-safe qualification harness passed independent offline review through its pre-uninstall boundary. The next reviewed run must prove its post-#9608 uninstall and final-absence path.
- PR #9795 adds ACP package and import checks to managed Hermes images. PR #9822 adds one bounded retry for a throttled Hermes source-archive download. These changes do not activate ACP support.

## Current status — 2026-08-24

GitHub-Verified `main` is `ffb09d2596fb2658f8dff881d5e1e425dd1767d6` with verification reason `valid` at this update.

Experimental activation remains blocked. No complete field result exists for this `main` commit.

Merged PR #10006 removed `--chmod` from checksum-pinned remote `ADD` instructions. Historical Brev evidence at `0e9fd51c493679cdbd513e17e3f3dfa593aad46b` failed on that exact instruction. That result remains historical evidence and does not qualify current `main`.

PR #10023 is the active implementation candidate. At latest PR commit `bf852f6d3c97aaf77431d70009e19ae174bf2eed`, it:

- removes the remaining BuildKit-only `RUN` options from the Portable staged Dockerfile;
- rejects unsupported remote `ADD`, local `COPY`, and `RUN` options before context reservation;
- stages and builds the Hermes image in the protected rootless-Podman check;
- binds builder child processes to the qualified rootless Podman socket;
- settles OpenShell `Ready` publication under one total deadline;
- adds UID-agnostic rootless regression coverage.

At this update, GitHub reports `APPROVED` review state and `UNSTABLE` merge state. The `static-checks`, `build-typecheck`, and `cli-test-shards (8)` checks fail. Other checks remain in progress. Exact-candidate admission has not passed.

The next field activity is one non-qualifying diagnostic rehearsal on at most one new diagnostic VM after exact-candidate admission passes. It must exercise every applicable lifecycle checkpoint, group reachable sibling failures by root cause, and produce no qualification or support claim. Retained failed VMs remain evidence only and must not be reused.

Fresh GPU-enabled and `--no-sandbox-gpu` qualification runs may start only after the diagnostic rehearsal passes, the accepted correction merges, and both runs bind to one GitHub-Verified `main` commit.

## Current implementation boundary

Current source owners include:

- `src/lib/onboard/experimental/hermes-portable-receipt.ts` for schema-5 receipt parsing and publication;
- `src/lib/onboard/experimental/hermes-portable-onboarding.ts` for the enrollment transaction;
- `src/lib/onboard/experimental/hermes-portable-lifecycle.ts` and `portable-agent-lifecycle.ts` for receipt-owned lifecycle operations;
- `src/lib/onboard/experimental/hermes-portable-podman-authority.ts` for current-user rootless Podman authority;
- `src/lib/onboard/experimental/hermes-portable-policy-authority.ts` for effective-policy evidence;
- `src/lib/onboard/experimental/hermes-portable-build-context.ts` and `hermes-portable-build-context-files.ts` for build-context custody;
- `src/lib/onboard/experimental/hermes-portable-ollama-inference.ts` and its runtime-provider owners for managed Ollama startup and recovery;
- `src/lib/actions/uninstall/portable-runtime-cleanup.ts` and `src/lib/state/portable-uninstall-retirement.ts` for shared retirement admission;
- the schema-5 uninstall transaction delivered through PR #9903 for Hermes-owned sandbox, provider, inference, receipt, and registry retirement.

The implementation supports the complete schema-5 lifecycle. The accepted matrix still requires protected field evidence before activation.

## Architecture and security boundaries

- Preserve schemas 1 through 4 as OpenClaw receipts. Schema 5 is agent-discriminated Hermes authority.
- Bind `agent=hermes`, phase, socket identity, full container ID, required OpenShell labels, lifecycle generation, source and image fingerprints, startup descriptor, health descriptor, effective policy identity, and sandbox registry identity.
- Treat sandbox names as metadata. Never use a name alone as mutation authority.
- Reject an unknown schema, agent, or phase; duplicate receipts; short IDs; replaced sockets; duplicate or missing containers; label or generation drift; and receipt-registry-live disagreement before mutation.
- Do not use Docker discovery, Docker GPU patching, Docker orphan cleanup, ambient `DOCKER_HOST`, or Docker fallback for a Podman-owned Hermes container.
- Use the manifest-owned startup command, port 8642 authenticated `/health`, `device_pairing=false`, and `/sandbox/.hermes` state.
- Hold one lifecycle lock across each authorized state transition. Revalidate authority before every mutation.
- Publish receipt and policy files with bounded UTF-8, private ownership and mode, no symlinks, no hard links, `O_NOFOLLOW`, durable replacement, and crash-safe recovery.
- Keep credentials, Bearer tokens, pairing state, command output, and raw policy material out of logs and artifacts.
- Keep dashboard observations separate from probe, health, TUI, chat, and inference acceptance.
- Do not edit administrator-owned systemd or cgroup configuration, invoke automatic `sudo` remediation, weaken CPU limits, or write cgroup files.

## Remaining capability slices

### Slice 2: Schema-5 receipt-owned retirement and uninstall — delivered

Owner: #9608, delivered through PR #9903.

Outcome: NemoClaw has deterministic schema-5 uninstall and recovery coverage. PR #9870 separately preserves Bedrock adapter evidence when shutdown fails.

Remaining evidence: exercise the delivered transaction in the protected field journey. Prove interruption recovery, a second no-op uninstall, final absence, and preservation of unrelated Docker and Podman resources.

### Slice 3: Current-main Hermes proof of life

Outcome: fresh dedicated Brev VMs verify the public Hermes Portable path on one GitHub-Verified `main` commit.

Required evidence:

- reviewed host preparation, real reboot, and post-reboot admission;
- current-user Podman authority and unchanged unrelated Docker state;
- onboarding 8/8 and OpenShell `Ready`;
- schema-5 receipt, registry, socket, container, labels, lifecycle generation, and fingerprints agree;
- `nemohermes connect --probe-only` exits 0;
- `nemohermes launch ` reaches the reviewed Hermes TUI marker;
- authenticated `/health`, one prompt-correlated response, and one real inference request pass as separate facts;
- repeated launch timing is recorded as measurement only;
- stop and start preserve the full container ID and Hermes state;
- a real host reboot requalifies the socket and recovers the same receipt-owned container;
- full uninstall removes only receipt-owned Hermes resources;
- a second completed uninstall performs no mutation and exits successfully;
- final absence checks preserve unrelated Docker and Podman resources;
- one L40S GPU-enabled run and one separate `--no-sandbox-gpu` run pass.

### Slice 4: Activation decision

Outcome: maintainers review the complete evidence and record whether the experimental capability can be activated.

Required evidence:

- independent nine-category security review binds to the tested implementation;
- documentation writer review binds to the final user-facing wording;
- CI, image provenance, Portable Profile, and protected field evidence are accepted or have recorded dispositions;
- maintainers record the final experimental wording and activation decision on this Epic.

## Delivery order

1. Resolve the failing and pending checks on the exact latest PR commit for #10023, or accept a replacement correction.
2. Run one non-qualifying diagnostic rehearsal through every applicable lifecycle checkpoint. Do not use a retained failed qualification VM.
3. If the rehearsal finds a defect, inspect all sibling paths in the same state transition. Use one coherent correction for each root-cause group. Do not open one PR for each field symptom.
4. Merge #10023 and any accepted diagnostic correction. Reread GitHub-Verified `main` and complete the narrow composition review.
5. Obtain acceptable current-main CI, Hermes image, and Portable Profile evidence. Record separate dispositions for unrelated platform failures.
6. Run the complete L40S GPU-enabled journey on one fresh dedicated VM.
7. Run the separate `--no-sandbox-gpu` journey on another fresh evidence identity.
8. Record security, documentation, and maintainer activation decisions.

A new `main` commit invalidates an unstarted evidence identity. A latest PR commit change invalidates unstarted diagnostic evidence. A semantic overlap requires a new narrow review before mutation.

## Activation gates

- [x] #9188 establishes fail-before-effects CPU admission through every required current-user cgroup boundary.
- [x] #9203 and PR #9424 implement the core schema-5 receipt, onboarding, start, stop, connect, probe, and recovery contract.
- [x] The production-shaped installer-to-active regression passes through PR #9579.
- [x] #9577 and PR #9578 establish network-before-alias ordering.
- [x] #9587, PR #9641, and PR #9665 establish non-overlapping host-gateway authority and fail-closed retired-alias evidence.
- [x] #9596 and PR #9632 establish receipt-owned managed Ollama under current-user Podman.
- [x] PR #9635 preserves Hermes Portable exclusion from stock managed-runtime activation.
- [x] The replay-safe protected qualification harness passed independent offline review.
- [x] PR #9790 aligns Hermes Portable with OpenShell 0.0.106.
- [x] PR #9694 pins rootless Podman, and PR #9714 establishes the Portable subnet and retired-subnet recovery.
- [x] PR #9837 binds protected Hermes image selection to the reviewed base-image index.
- [x] #9608 and PR #9903 implement schema-5 receipt-owned retirement and uninstall.
- [x] PR #9870 makes Bedrock adapter shutdown fail closed before evidence retirement.
- [x] PR #10006 removes the historical remote `ADD --chmod` incompatibility and preserves checksum-pinned acquisition.
- [ ] #10007 and PR #10023, or an accepted replacement, remove remaining BuildKit-only `RUN` usage and pass exact-candidate admission.
- [ ] One non-qualifying diagnostic rehearsal crosses the complete applicable lifecycle before fresh qualification.
- [ ] #9921 is closed or explicitly dispositioned after the accepted image-build correction lineage lands.
- [ ] Current-main CI/Main, Platform Compatibility, Portable Profile, and applicable Hermes image evidence are accepted or have recorded dispositions.
- [ ] A fresh current-main L40S journey passes install, onboarding, probe, TUI, chat, inference, stop, start, reboot recovery, uninstall, and second uninstall.
- [ ] A separate `--no-sandbox-gpu` journey remains Podman-owned and passes the same applicable lifecycle gates.
- [ ] Shared Personal policy, Hermes Bearer authentication, and Hermes pairing contracts pass in protected evidence.
- [ ] Full uninstall removes only receipt-owned Hermes resources and preserves unrelated resources.
- [ ] Every final implementation PR has DCO, GitHub verification, required CI, review resolution, documentation writer review, and required reviewer approval.
- [ ] Maintainers explicitly accept the final experimental boundary and documentation wording.

## Constraints and non-goals

- Do not add this Epic or #9203 under OpenClaw Epic #9200.
- Do not activate or claim the production Podman provider.
- Do not claim general Linux, architecture, GPU, model, inference, or Docker compatibility beyond the accepted matrix.
- Do not create PRs in OpenShell, OpenClaw, or Hermes repositories through this Epic.
- Do not use dashboard readiness as the TUI or chat oracle.
- Do not reuse a disqualified qualification VM.
- Do not mutate a host that has not passed the admission gate.
- Do not infer product support from technical compatibility, deterministic tests, or one field run.

## Work tracking

- [x] Set native Issue Type to `Epic` and add this issue to Project 199.
- [x] Complete the core lifecycle implementation through PR #9424.
- [x] Merge the installer-to-active regression and fixture-custody correction through PR #9579.
- [x] Resolve #9577 through PR #9578.
- [x] Resolve #9587 through PR #9641 and PR #9665.
- [x] Resolve #9596 through PR #9632.
- [x] Preserve Hermes Portable exclusion from stock managed-runtime activation through PR #9635.
- [x] Complete independent offline review of the replay-safe qualification harness.
- [x] Merge the OpenShell 0.0.106 alignment through PR #9790.
- [x] Merge the pinned rootless Podman and Portable subnet corrections through PR #9694 and PR #9714.
- [x] Resolve managed Hermes base-image selection through PR #9837.
- [x] Implement schema-5 uninstall through #9608 and PR #9903.
- [x] Correct the separate adapter evidence-loss path through PR #9870.
- [x] Merge PR #10006 to remove the historical remote `ADD --chmod` incompatibility.
- [ ] Complete #10007 through PR #10023 or an accepted replacement.
- [ ] Run one non-qualifying diagnostic rehearsal and batch same-root-cause findings before fresh qualification.
- [ ] Close or explicitly disposition #9921 after the accepted image-build correction lineage lands.
- [ ] Accept or disposition current-main CI/Main, Platform Compatibility, Portable Profile, and Hermes image evidence.
- [ ] Record the complete GPU-enabled Brev journey.
- [ ] Record the separate `--no-sandbox-gpu` journey.
- [ ] Record final security, documentation, and maintainer activation decisions.

## Category

Platform

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.