NVIDIA / NVIDIA/NemoClaw

Retire NemoClaw-owned Docker and Podman lifecycle orchestration

Open
#11,905 0 comments 0 reactions 0 assignees View on GitHub
area: architecture area: e2e area: onboarding area: sandbox enhancement needs: triage platform: container
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Problem Statement

The accepted basic-onboarder direction in #11255 is:

1. Select the supported agent and immutable image.
2. Create the OpenShell sandbox.
3. Place credentials as required by OpenShell and attach the initial non-secret inference route.
4. Launch the native agent and return connection details.
5. Leave the ongoing control path.

#11765 and #11772 returned native gateway/background-process lifecycle to the agent and ordinary sandbox lifecycle to OpenShell. PRs #11792 and #11776 removed much of the agent-gateway controller and obsolete recovery machinery.

Current `main` still contains a second container-runtime lifecycle system around standard managed-image Docker and native Podman onboarding:

- provider-specific Docker and Podman start, stop, recover, inspect, and verification paths;
- a runtime-provider lifecycle surface and mutation authority for ordinary sandbox lifecycle;
- Podman hold/swap/watcher leases and durable replacement journals;
- shared Docker/Podman managed-bootstrap cutover, rollback, and finalization machinery;
- lifecycle-specific E2E setup, fixtures, workflow routing, and assertions.

This duplicates behavior already owned by OpenShell and conflicts with the basic-onboarder end state.

At `main` commit `7c776cdfa1c4928b641d00bddd5d57d113557ae2`:

- #9923 and #11014 added a combined net 19,596 tracked-text lines for native Podman activation and Docker-independent lifecycle support.
- The current Podman-specific bootstrap, lifecycle, readiness, and qualification cluster is approximately 16,203 lines.
- `src/lib/onboard/managed-bootstrap/` is 27,666 lines: 17,046 production and 10,620 tests.
- The expected deletion opportunity for the standard managed-image Docker/Podman paths is approximately 30,000–35,000 production, test, and workflow lines.

The LOC estimate is planning evidence, not authorization to weaken behavior to hit a number. Every retained contract must have a current consumer.

## Product Decision and Outcome

**Decision: Accept.**

- **Reason:** remove duplicate lifecycle ownership and complete the accepted basic-onboarder architecture.
- **Placement:** core NemoClaw.
- **Accountable owner:** NemoClaw onboarding/runtime maintainers.
- **Validation:** one atomic PR must preserve successful standard Docker, native Podman, and Portable outcomes on the same exact head while deleting the obsolete lifecycle implementation and qualification surface.

Standard managed-image Docker and native Podman onboarding use this ownership model:

```text
supported agent + exact managed-image digest
|
v
OpenShell sandbox creation
|
v
credential placement + initial route/reference
|
v
native agent launch
|
v
connection details; NemoClaw exits
```

OpenShell owns sandbox start, stop, deletion, containment, runtime reconciliation, and Docker/Podman driver behavior. The native agent owns its gateway, cron, child agents, hooks, subprocesses, packages, plugins, updates, and background work.

NemoClaw retains only:

- supported-agent and exact managed-image selection;
- minimum host prerequisite and OpenShell driver selection for initial creation;
- credential placement and projection required by OpenShell;
- initial policy, provider, and non-secret inference-route attachment;
- exact-target authority at initial mutation and cleanup edges;
- bounded diagnostics and supported-platform qualification.

## Atomic Delivery Contract

Deliver this issue in **one atomic pull request against current `main`**.

The PR must contain, together:

- the standard Docker and native Podman production-code deletion;
- the final thin OpenShell-owned lifecycle and exact-image creation path;
- migration or compatibility handling required for already installed supported standard sandboxes;
- deletion or refocusing of obsolete unit, integration, E2E, workflow, fixture, and helper code;
- documentation and command-reference updates;
- exact-head deterministic, CI, review, and runtime evidence;
- a before/after inventory of production LOC, test/workflow LOC, and E2E jobs.

Do not land preparatory refactors, donor PRs, temporary adapters, intermediate registries, compatibility modes, or follow-up cleanup PRs for this issue. The repository must not pass through a state with both old and new lifecycle authorities.

If the pinned OpenShell create result lacks sufficient immutable resource identity, the same PR may retain one narrow exact-create/cleanup receipt. It must not retain the replacement-container controller as a workaround. If safe atomic delivery requires an OpenShell change, stop and record that dependency rather than splitting this issue or weakening exact-target cleanup.

The PR is not complete until Docker, native Podman, Portable regressions, CI, review, and required live evidence all pass on the same final head.

## User-Visible Behavior

Successful behavior remains compatible:

- Standard `nemoclaw onboard` commands and supported results remain unchanged.
- Supported agents, exact images, policies, credential placement, and initial inference routes remain unchanged.
- Public lifecycle commands remain available with their current exit-code and machine-readable contracts unless a separate accepted decision changes them.
- Workspace and native-agent state survive OpenShell stop/start as before.
- Destroy and uninstall retain their supported outcomes.
- Portable behavior is unchanged.

Intentional failure/recovery changes:

- OpenShell reports and owns sandbox lifecycle failures.
- Native agents report and own gateway/background-process failures.
- NemoClaw no longer silently inspects, restarts, replaces, repairs, or quarantines standard Docker/Podman containers during ordinary lifecycle operations.
- Ambiguous create or cleanup results are re-observed through exact OpenShell identity rather than entering a NemoClaw replacement/rollback controller.
- Diagnostics may use OpenShell/native terminology, but remain bounded, actionable, redacted, and compatible with documented machine-readable output.

## Scope and Exclusions

### Included

- Standard managed-image Docker onboarding.
- Qualified native Podman onboarding selected through the existing public runtime path.
- Remove direct Docker/Podman container start, stop, restart, recover, and lifecycle-state inference from ordinary sandbox lifecycle.
- Remove the runtime-provider lifecycle surface and ordinary `start`/`stop` mutation authority where OpenShell owns the operation.
- Replace Podman hold/swap/watcher/journal bootstrap with direct exact-image OpenShell creation plus the smallest required initial receipt.
- Apply the same simplification to shared Docker managed-bootstrap machinery whose only consumer is container lifecycle control.
- Remove lifecycle-only fixtures, targets, workflow jobs, and assertion matrices.
- Preserve public commands through thin OpenShell delegation where they remain supported.
- Record deleted production functions, tests, E2E jobs, and final net LOC.

### Hard Exclusions

- **Portable is retained.** Do not remove, weaken, generalize away, or route around Portable’s Docker-over-Podman path, local registry, receipts, lifecycle locks, recovery, or qualification. Portable remains a separate explicit experimental profile and is not the generic replacement for standard Podman.
- Explicit custom-Dockerfile, persisted legacy-Dockerfile, and custom-image behavior remain unchanged.
- MXC and Windows native-artifact behavior remain unchanged.
- Public lifecycle-command deprecation is not authorized.
- Credential placement, OpenShell containment, policy enforcement, exact-target privileged operations, and initial inference attachment remain.
- Broader snapshot/restore removal and host-local inference lifecycle simplification remain separate work under #11255.
- NVIDIA/OpenShell changes require separate explicit approval.
- Do not add a replacement NemoClaw lifecycle abstraction, registry, journal, fallback, or compatibility mode solely to preserve retired behavior.

## Acceptance Criteria

### Ownership and source shape

- [ ] Standard Docker and Podman sandbox lifecycle operations execute through OpenShell, not direct container mutation.
- [ ] No standard-path source invokes `docker start|stop|restart` or `podman start|stop|restart` for sandbox lifecycle.
- [ ] No standard-path source treats raw container state as authoritative OpenShell lifecycle state.
- [ ] The standard runtime-provider bundle no longer exposes a second lifecycle controller or ordinary `start`/`stop` mutation authority.
- [ ] Podman watcher leases, held-workload replacement, image transactions, cutover journals, and rollback controllers have no standard-path consumer and are deleted.
- [ ] Equivalent shared Docker managed-bootstrap lifecycle machinery is deleted where direct exact-image creation replaces it.
- [ ] No new lifecycle abstraction, registry, fallback, compatibility mode, or duplicated source of truth replaces the removed code.
- [ ] The completed PR is net-negative in production, tests, and workflow support.

### Allowed behavior

- [ ] Standard Docker onboarding succeeds for OpenClaw, Hermes, and LangChain Deep Agents Code with the selected exact managed image.
- [ ] Native Podman onboarding succeeds for the supported Linux amd64/arm64 cohort with Docker unavailable.
- [ ] OpenShell stop/start preserves workspace/native-agent state and returns the agent to readiness.
- [ ] Supported public lifecycle commands retain their documented exit codes and machine-readable output.
- [ ] Destroy/uninstall removes only the exact OpenShell-owned resource.
- [ ] Portable’s existing focused qualification remains unchanged and passes.

### Denied and ambiguous behavior

- [ ] Wrong provider, gateway, sandbox identity, image digest, or stale receipt fails before mutation or cleanup.
- [ ] A same-name resource is never accepted as ownership evidence.
- [ ] An ambiguous post-create result assumes the resource may exist, re-reads exact OpenShell state, and does not blindly create or delete.
- [ ] Cleanup failure or ambiguity preserves the non-secret exact identity and stops without name-only retry.
- [ ] Standard-path runtime code cannot fall back to the retained Portable controller.
- [ ] Credentials remain absent from argv, journals, logs, diagnostics, artifacts, and model-visible output.

### Failure and recovery behavior

- [ ] Failure before any remote effect publishes no ownership.
- [ ] Verification failure does not report onboarding success.
- [ ] Exact cleanup success proves resource absence through OpenShell.
- [ ] Native/OpenShell failures remain bounded, actionable, and redacted without triggering a NemoClaw lifecycle controller.
- [ ] Existing supported standard sandboxes have an explicit compatible transition or bounded unsupported-state diagnostic.
- [ ] No interrupted legacy transaction is discarded when exact ownership or absence cannot be proven.

## Validation Plan

### Deterministic validation

Use current repository owners rather than creating a parallel suite:

- runtime-provider contract and registration checks;
- exact image-selection and workload-receipt tests;
- typed OpenShell create/start/stop/delete adapter tests;
- wrong-owner, stale-identity, name-reuse, interrupted-create, and interrupted-cleanup regressions;
- credential-canary and diagnostic-redaction regressions;
- Portable contract and lifecycle-lock regressions;
- explicit custom/legacy Dockerfile regressions;
- repository architecture, source-shape, generated-contract, typecheck, lint, and package checks.

### Runtime and E2E validation

Refocus existing targets:

- standard Docker onboarding for the shipped agents;
- native Podman onboarding with Docker unavailable;
- exact managed-image and startup-contract observation;
- credential placement, containment, and negative-egress evidence;
- OpenShell stop/start plus native-agent readiness and state survival;
- native cron, hooks, child agents, and background work without NemoClaw authorization or repair;
- exact destroy/uninstall cleanup;
- bounded failure and cleanup evidence;
- unchanged Portable qualification.

The PR must record the removed targets/jobs and the reduced fanout. Do not create a second native-behavior matrix.

### Quantitative evidence

The PR description must report:

- starting and final `main`/PR head;
- tracked-text additions, deletions, and net change;
- production versus test/workflow change;
- removed E2E targets and jobs;
- every retained Docker/Podman-specific module and its current consumer;
- why any result materially below the current 30,000–35,000-line estimate was required for correctness, security, or compatibility.

## Compatibility Requirements

Preserve:

- the supported standard Docker cohort;
- qualified native Podman Linux amd64/arm64;
- OpenClaw, Hermes, and LangChain Deep Agents Code managed-image onboarding;
- the repository-pinned OpenShell contract;
- exact image-digest selection;
- OpenShell credential custody, policy, containment, and sandbox identity;
- Portable without behavioral reduction;
- explicit custom and legacy Dockerfile behavior;
- supported destroy, uninstall, and initial-failure cleanup.

No new platform, agent, provider, image, or lifecycle support is introduced.

## Security or Privacy Impact

This change removes duplicate security-sensitive lifecycle authority; it does not weaken OpenShell controls.

- OpenShell remains authoritative for sandbox containment and lifecycle.
- Native agents remain authoritative for their gateways and background processes.
- NemoClaw credential placement remains limited to the OpenShell-required boundary.
- Initial resource mutation and cleanup remain bound to exact gateway, sandbox, provider, and image identity.
- Same-name resources never establish authority.
- Transport ambiguity is treated as a possible remote effect.
- Credential values remain in host/OpenShell custody for their existing lifetime and are removed through the current OpenShell/provider mechanism.

Security evidence must cover wrong-owner, stale identity, name reuse, interrupted creation, interrupted cleanup, credential canaries, and Docker-unavailable Podman.

## Related Work and Coordination

- #9923 and #11014 introduced the native Podman activation and Docker-independent lifecycle work being simplified.
- #11792 and #11776 established the new native-agent/OpenShell lifecycle boundary.
- Open PR #11838 touches the lifecycle-adapter and deletion boundary. Incorporate its applicable work into the atomic PR or wait for it to land; do not create a competing adapter.
- Open PR #11877 changes managed-bootstrap Docker runtime behavior. Reconcile it before removing that owner.
- Open PRs #11614 and #11871 expand lifecycle/recovery diagnostics that may be superseded. Reassess them rather than porting their logic into the replacement.
- Portable-related work, including #11874 and #11887, remains outside this issue and must not be blocked, folded in, or deleted.
- Refresh `origin/main`, the pinned OpenShell version, and overlapping PR states before implementation begins.

## Ongoing Ownership

Core NemoClaw onboarding/runtime maintainers own the resulting thin OpenShell integration.

One implementation owner and one atomic PR should be identified when work begins.

## Placement and Support Expectations

Placement: core NemoClaw.

Support remains unchanged for successful standard Docker, qualified native Podman, and Portable usage. The implementation changes lifecycle ownership and failure handling, not the supported success outcomes.

## Category

Platform

## Checklist

- [x] I searched existing issues and found no duplicate.
- [x] I described the problem and desired behavior.

Contributor guide

Open the contributing guide

Research direction

Start by inventorying standard-path consumers under src/lib/onboard/managed-bootstrap/ and the runtime-provider contract and registration checks. Run the existing exact-image, workload-receipt, typed OpenShell adapter, Portable, custom-Dockerfile, architecture, typecheck, lint, and package checks named in the validation plan. Done means one atomic PR leaves OpenShell owning standard Docker and Podman lifecycle, preserves Portable and public contracts, and removes obsolete lifecycle tests, workflows, fixtures, and production code.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
devops, infrastructure
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.