[Epic] Deliver the Express Installer for DGX Spark and DGX Station

## Outcome

NemoClaw provides a supported, low-friction Express installation experience on DGX Spark and DGX Station. The standard installer detects qualified hardware, offers the existing Express `[Y/n]` path, selects safe platform defaults, and supports discoverable secondary serving profiles without adding one CLI flag per model.

This epic owns the Q3 product surface and release evidence for Express on both platforms. It does not make every Spark or Station issue a dependency.

## Product decisions

- Keep the standard installer and hardware-detected Express prompt; do not add a separate Express entry point.
- Keep `nvidia/Qwen3.6-35B-A3B-NVFP4` as the Spark Express default until an evidence-backed product decision changes it.
- Offer public alternative models, including NVIDIA Nemotron 3.5 Lightning NVFP4, as explicit secondary profiles after their immutable artifacts and serving contracts pass qualification.
- Use a generic, discoverable profile mechanism such as `nemoclaw profiles list` and `--profile `.
- Do not add model-specific flags such as `--spark-lightning`. Preserve compatibility aliases during migration where required.
- Keep model, image, parser, frontend, environment, and launch settings in repository-owned serving recipes and presets.
- Treat profile availability, automatic default selection, and advertised platform support as separate decisions.

## Current platform experience

### DGX Spark Express

When accepted, the effective default remains:

- Provider: managed local vLLM
- Model: `nvidia/Qwen3.6-35B-A3B-NVFP4`
- Agent: OpenClaw
- Sandbox: `my-assistant`
- Policy: suggested mode with the balanced tier

Secondary profiles must be explicitly selected and must not alter this default implicitly.

### DGX Station Express

Preserve the existing qualified Station Express behavior while migrating model selection into the same catalog and profile UX. Existing compatibility paths such as `--station-deepseek` may remain as aliases during a documented migration, but new model-specific flags must not be added.

## DGX Spark product boundary

DGX Spark Express has two separately gated paths:

- Single Spark keeps `nvidia/Qwen3.6-35B-A3B-NVFP4` as the automatic default.
- NVIDIA Nemotron 3.5 Lightning NVFP4 is an explicit-only secondary profile until qualification and product approval permit promotion.
- The automatic two-Spark profile introduced by merged PR #8129 remains **Experimental** until protected physical two-system E2E passes.

The distributed Spark path owns deterministic compatible-peer selection, direct ConnectX-7 topology, pretrusted SSH, ranked-node launch, exact plan/receipt identity, rollback, recovery, and uninstall. If no implicit peer qualifies, onboarding must fall back to the single-Spark default before distributed image or model downloads. Explicit profile or legacy inference intent remains authoritative and must never be silently replaced by automatic distributed selection.

Physical two-node activation evidence is still required; merged implementation alone is not a support claim.

## DGX Station product boundary

DGX Station is currently **Tested with limitations** for qualified single-Station GB300 profiles. Dedicated CI is absent, the accepted no-OTA DGX OS `7.6.x` family still needs full Express E2E qualification, and the automatic dual-Station path remains **Deferred**.

The Station deliverable includes these distinct paths:

- Single Station with the Nemotron 3 Ultra managed-vLLM Express default.
- DeepSeek V4 Flash as an explicit secondary choice, migrated from `--station-deepseek` to the generic profile UX while preserving a compatibility alias.
- Automatic trusted-pair discovery for Nemotron Ultra, with safe single-Station fallback when no implicit peer qualifies.
- An explicitly requested peer that must qualify or stop; it must never silently fall back.
- Supported manual-provider fallback when the operator declines Express or preserves a conflicting existing inference workload.

Qualified Station host families currently include generic Ubuntu 24.04 ARM64, accepted OTA-form DGX OS releases, accepted no-OTA DGX OS `7.6.x`, and exact qualified Colossus BaseOS and AI Developer Tools images. The shipped platform matrix must name only host families that have matching physical evidence.

Station host preparation is an action-safety-gated product surface, not merely a model launch. Depending on the host profile it may validate or change the NVIDIA driver, Docker/Buildx, NVIDIA Container Toolkit, CDI, service state, Docker-group membership, and reboot/login state. It must preserve unrelated Docker records, stop before a conflicting mutation, roll back failed runtime changes, and resume the exact accepted revision and selections.

The distributed path additionally owns reciprocal Station/GPU identity, two private `/30` ConnectX-8 rails, MTU/route/neighbor/MAC validation, pre-existing SSH trust, passwordless remote preparation, exact GPU and `uverbs` devices, unauthenticated Ray/NCCL trust disclosure, bearer-protected serving, rollback, and receipt-owned cleanup.

## Scope

### Detection and Express UX

- Detect supported Spark and Station hardware through the common readiness producer.
- Offer the existing Express prompt with an accurate configuration summary and download estimate before side effects.
- Provide a discoverable list of compatible profiles and a generic explicit selector.
- Reject unknown, disabled, ambiguous, incompatible, or unqualified profiles before downloads or host mutation.
- Clearly distinguish the automatic default from experimental or explicit-only alternatives.

### Serving catalog and model profiles

- Use the accepted serving-catalog design from Discussion #7636.
- Complete host-local single-Spark vLLM preset integration from #8246 and merged PR #8249.
- Represent Spark and Station defaults and alternatives as immutable recipes and presets.
- Pin the public model ID, revision, runtime image digest, frontend, parsers, preparation, launch arguments, and readiness contract.
- Add the public Lightning NVFP4 Spark profile only after direct-runtime and Express E2E evidence passes.
- Preserve the current automatic defaults unless a separate product decision changes them.

### Lifecycle, resume, and coexistence

- Persist preset ID, recipe ID, catalog digest, resolved-plan identity, selection source, and non-secret overrides.
- Make resume, status, diagnostics, recovery, uninstall, and cleanup consume the persisted identity.
- Re-run non-resume-safe readiness checks and never silently select a different default on resume.
- Handle pre-existing OpenShell installations and version differences safely: align/reuse when supported or stop before mutation with actionable remediation.
- Preserve independently managed OpenShell gateways and their sandboxes; the verified coexistence behavior from closed issue #8048 must not regress.
- Keep Hugging Face credentials out of command output, logs, receipts, and sandbox state, and provide actionable cache ownership/authentication remediation.

### Quality and release evidence

Qualify each supported platform/default/profile combination on physical hardware. Evidence must bind:

- NemoClaw commit/tag
- host/OS/driver and OpenShell versions
- exact model revision and runtime image digest
- complete serving command and catalog identities
- startup/download time
- loaded model, KV-cache, CUDA-graph, and host-memory measurements
- throughput and latency
- kernel/NVIDIA allocation errors
- chat and tool-use behavior
- cleanup and baseline restoration

## Acceptance criteria

### User experience

- [ ] The standard installer detects qualified DGX Spark and DGX Station hardware and offers the existing Express `[Y/n]` flow.
- [ ] Accepting Express produces the documented provider, model, agent, sandbox, and policy defaults.
- [ ] `nemoclaw profiles list` (or an accepted equivalent) shows compatible profile IDs, display names, support state, selection mode, model, and estimated download.
- [ ] A generic `--profile ` selector (or an accepted equivalent) works in interactive and non-interactive onboarding.
- [ ] No new model-specific installer flag is required for a secondary profile.
- [ ] Unknown or incompatible profiles fail before download, route, gateway, provider, or sandbox mutation.
- [ ] The final review screen names the selected profile, model, image source, support state, and download estimate.

### Defaults and optional profiles

- [ ] Spark Express remains on the current Qwen default when no profile is explicitly selected.
- [ ] Station Express retains its current qualified default and compatibility behavior.
- [ ] The public Lightning NVFP4 profile is explicit-only until separate evidence and product approval promote it.
- [ ] Every shipped profile uses an exact public model revision and immutable runtime image digest.
- [ ] Profile settings reproduce the approved deployment recipe without installer-specific duplication.
- [ ] Changing a default is a data/configuration change with one-day turnaround after approval, without adding a new CLI flag.

### DGX Spark qualification and lifecycle

- [ ] Single-Spark Express preserves the Qwen default when no profile is explicitly selected.
- [ ] Lightning is selectable only through the generic explicit profile path until its activation gate passes.
- [ ] Automatic dual-Spark selection occurs only after deterministic peer and topology qualification; no-match falls back before distributed downloads.
- [ ] Explicit or legacy inference intent is authoritative and is never silently replaced by automatic distributed selection.
- [ ] Direct ConnectX-7 topology, pretrusted SSH, ranked-node launch, receipt provenance, rollback, recovery, and uninstall are covered by protected physical two-node E2E.
- [ ] Dual Spark remains `Experimental` until the physical activation gate and support-matrix review pass.

### Functional validation

- [ ] Plain chat passes through the direct OpenAI-compatible API and through the Express-created OpenClaw sandbox.
- [ ] Automatic structured tool calling passes with the supported frontend and parser.
- [ ] An unhinted request to create a file and read it back succeeds without requiring knowledge of `tool_search -> tool_describe -> tool_call`.
- [ ] Named tool choice either works or is explicitly declared unsupported with tested behavior.
- [ ] Lightning passes the release workload: 32K ISL, 2K OSL, 1K added per turn, 10 turns, with recorded throughput and memory evidence.
- [ ] Default and secondary profiles pass cold start, cached restart, resume, recovery, cleanup, and baseline-restoration tests.
- [ ] The Spark physical test fails on relevant `NVRM`, `Xid`, `GSP`, `UVM`, or `NV_ERR` events, not only HTTP readiness.
- [ ] Performance and memory findings from #7127 are incorporated into default/profile qualification.

### Lifecycle and safety

- [ ] Resume preserves exact profile and recipe identity and does not silently fall back.
- [ ] Status and diagnostics report selected profile/recipe provenance and actionable readiness failures.
- [ ] Independently managed OpenShell gateways remain untouched.
- [ ] A pre-existing incompatible OpenShell version is safely aligned/reused or rejected before provider and sandbox mutation.
- [ ] Authenticated model acquisition does not expose credentials and handles cache ownership predictably.
- [ ] Failed onboarding leaves a resumable or cleanly recoverable state.
- [ ] Test cleanup restores the prior healthy inference container, shared route, gateway, and existing sandboxes.

### DGX Station qualification and lifecycle

- [ ] The platform matrix states the exact single-Station host profiles with physical evidence and keeps unqualified profiles out of support claims.
- [ ] Generic Ubuntu 24.04 ARM64, accepted OTA DGX OS, accepted no-OTA DGX OS `7.6.x`, Colossus BaseOS, and AI Developer Tools paths each have an explicit qualification result or are removed from the advertised matrix.
- [ ] Preflight verifies GB300 identity, driver, zero volatile ECC errors, Docker/Buildx, NVIDIA Container Toolkit, CDI, container GPU access, storage, services, Secure Boot and kernel prerequisites where applicable.
- [ ] Host preparation follows #7153: unrelated stopped containers are preserved, running workloads block only conflicting operations, and NemoClaw never rewrites or removes unrelated workloads.
- [ ] Docker/runtime/driver mutation and reboot boundaries re-check active workloads and auto-restarting stopped containers immediately before the disruptive action.
- [ ] Every failed host mutation either restores the prior configuration or reports exact bounded recovery state.
- [ ] Reboot and Docker-group login handoffs print one continuation command bound to the accepted revision, agent, profile, policy, sandbox, and ports.
- [ ] An existing vLLM workload produces a safe keep-and-resume or manual-validation choice without silently stopping it.
- [ ] Single-Station Express retains Nemotron Ultra as its default until an approved product change.
- [ ] DeepSeek remains selectable through the generic profile interface; `--station-deepseek` has a tested compatibility/deprecation path.
- [ ] Automatic dual-Station discovery examines only deterministic counterparts on the two configured private `/30` rails and never scans or enrolls trust.
- [ ] An implicit peer no-match falls back before image/model download; an explicit peer mismatch fails before effects.
- [ ] Dual-Station qualification binds reciprocal host, GPU, rail, route, neighbor, MAC, jumbo-frame, SSH-host-key, controller-account, and topology identities.
- [ ] Distributed Ray/NCCL/vLLM traffic, host networking, firewall responsibility, bearer-key handling, and trusted-network requirements are disclosed before effects and validated.
- [ ] Distributed containers use the accepted non-root identity, exact GPU and `uverbs` devices, read-only root/model cache, dropped capabilities, and `no-new-privileges`.
- [ ] Dual-Station startup is rollback-safe and writes an owner-only cleanup receipt; uninstall removes only the exact qualified pair.
- [ ] Same-revision reruns reuse the installed CLI and healthy managed inference without rebuilding, restarting, or backing up unnecessarily.
- [ ] OpenShell CLI/gateway version skew is detected before route mutation and is aligned, reused, or rejected with actionable remediation.
- [ ] The Nemotron Ultra recipe has an accepted FP8 KV-scale correctness boundary and a qualified GB300 Mamba SSU configuration.
- [ ] Profile/provider switching reconciles the route and the selected agent's in-sandbox model binding atomically, or leaves the old working configuration unchanged.
- [ ] Named sandboxes resolve their registered gateway rather than mutable global gateway selection during status, exec, recovery, and cleanup.
- [ ] Protected E2E covers OpenClaw plus every additional agent advertised for Station Express; agent/model limitations are reported separately from installer defects.
- [ ] Single Station remains `Tested with limitations`, and dual Station remains `Deferred`, until their respective activation gates pass.

### Cross-cutting installer compatibility

- [ ] The future public npm/npx entrypoint preserves the same Express prompt, resolved plan, profile selection, and resume behavior without duplicating Spark/Station logic.
- [ ] Users can review and correct configuration before the first mutation.
- [ ] Gateway-port intent persists across re-onboarding.
- [ ] Rebuilds preserve a consistent OpenShell shields/security state.
- [ ] Agent/model-specific failures are reported separately from platform-installer failures.

### Release readiness

- [ ] Unit, installer-integration, package-contract, and protected physical E2E coverage pass.
- [ ] The support matrix distinguishes automatic, explicit-only, experimental, and unsupported profiles.
- [ ] Documentation covers Express selection, profile discovery, non-interactive use, authentication, disk/memory requirements, recovery, and troubleshooting.
- [ ] Public documentation contains no private checkpoint names, internal paths, credentials, or prerelease details.
- [ ] A documentation writer and platform owners review the final experience and evidence.

## Work tracking

- [x] Discussion #7636 — accepted declarative serving recipe and preset architecture.
- [x] PR #8249 — model-independent host-local vLLM preset foundation.
- [x] PR #8129 — experimental automatic dual-Spark Express implementation.
- [ ] #8246 — complete single-host Spark lifecycle, resume, provenance, status, and cleanup requirements.
- [ ] #8384 — add discoverable serving profiles and generic `--profile ` selection with compatibility handling.
- [ ] #8385 — add the immutable public Lightning NVFP4 Spark recipe and explicit-only preset after qualification.
- [ ] Migrate existing Spark and Station Express defaults/alternatives to the catalog without changing behavior.
- [ ] Converge readiness with existing Spark/Station host-preparation and storage-remediation paths.
- [ ] Qualify pre-existing OpenShell version alignment, reuse, coexistence, and fail-before-effects behavior.
- [ ] #8386 — add protected physical single/dual-Spark E2E and activation evidence.
- [ ] #8382 — add protected physical Station E2E for advertised hosts, profiles, topologies, and agents.
- [ ] Publish profile/support documentation and release evidence.
- [x] Incorporate completed single-Spark Lightning validation findings into #8385, #8386, #8384, and #8246.

### DGX Station tracking

- [x] PR #6883 — add single-Station Nemotron Ultra Express.
- [x] PR #7030 — add automatic trusted dual-Station preparation and runtime.
- [x] PR #7059 — preserve Station Express choices on resume.
- [x] #7153 — accept action-safety-gated bounded coexistence on non-empty Docker hosts.
- [x] PR #7155 — preserve stopped Station containers.
- [ ] #8093 — close the OpenShell CLI/gateway version-skew route failure with current physical evidence or a fail-before-effects fix.
- [ ] #7898 — verify same-revision Express rerun reuse on the release containing PR #8316.
- [ ] #7899 — establish and qualify the Nemotron Ultra FP8 KV-scale correctness boundary.
- [ ] #7900 — ship or accept a qualified GB300 Mamba SSU configuration.
- [ ] #7994 — add deterministic recovery and externally supervised gateway test seams.
- [ ] #7105 — resolve sandbox-to-gateway ownership for status, exec, recovery, and cleanup.
- [ ] #7011 — make OpenClaw model/provider switching reconcile the in-sandbox binding.
- [ ] #7083 — make Hermes route switching atomic or preserve the prior route.
- [ ] #8381 — migrate Ultra single/dual and DeepSeek to declarative profiles, generic selection, compatibility aliases, and provenance.
- [ ] #8382 — add protected Station E2E for each advertised host profile, topology, serving profile, supported agent, rerun, resume, rollback, and cleanup.
- [ ] Update the platform matrix only from exact physical evidence; do not promote dual Station or no-OTA `7.6.x` prematurely.

## Cross-cutting coordination (linked, not owned as child deliverables)

- #7407 — common host-readiness producer used by Spark and Station detection.
- #8377 — public npm/npx installer replacement must preserve the same Express decisions and resolved plan without duplicating platform logic.
- #6005 and PR #8171 — configuration review/back-navigation must allow correction before side effects.
- #7791 — a selected gateway-port override must persist correctly across re-onboarding.
- #7985 — rebuild and OpenShell shields/security state must remain consistent under the cross-platform race.
- PR #7463 — Station-supported DCode/Ultra agent template compatibility.
- #7104, #8132, and #8133 — Hermes/model capability and agent-exit evidence; these are not installer-owned defects.

These items can block a release combination or supply evidence, but remain owned by their respective cross-platform, agent, or model workstreams.

## Related evidence and dependencies

- Discussion #7636 — accepted declarative managed-inference catalog design.
- #8246 / merged PR #8249 — host-local single-Spark serving preset foundation.
- #7127 — Spark managed-vLLM memory, MTP, long-context, and kernel-event qualification.
- #7304 — natural-prompt tool discovery behavior with the current Spark default.
- #8385 / #8386 — completed public Lightning single-Spark runtime, Express/OpenClaw, tool-use, workload, restart, cleanup, and restoration evidence; dual Spark was not tested.
- Closed #8048 — verified independent OpenShell gateway coexistence; retain as regression evidence.

## Delivery target

Target Q3. Public activation of any profile remains gated by immutable artifacts, product approval, documentation, and physical hardware evidence.

## Ownership

- Product/feature owner: Prekshi Vyas
- Platform collaborators: DGX Spark and DGX Station owners
- Inference/model collaborators: managed-inference and model-builder owners
- Agent/tool-use collaborators: OpenClaw and progressive tool-disclosure owners

## Category

Feature

## Checklist

- [x] Existing issues, PRs, and the accepted serving-catalog discussion were reviewed.
- [x] The current defaults and optional-profile decision are recorded.
- [x] Private and prerelease checkpoint details are excluded.