NVIDIA / NVIDIA/NemoClaw

[DeepSeek Harness 7/8] Qualify the exact candidate on AMD64 and ARM64

Open
#9,335 0 comments 0 reactions 0 assignees View on GitHub
area: ci area: e2e integration: deepseek-harness
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

Qualify one exact DeepSeek Harness candidate on fresh `linux/amd64` and `linux/arm64` environments with deterministic tests and live evidence for every product surface, lifecycle operation, inference route, state rule, and security boundary accepted by Gates 1–6.

Parent epic: #9328

Depends on completion of:

- #9329, **[DeepSeek Harness 1/8] Accept product scope, ownership, threat model, and compatibility matrix**;
- #9330, **[DeepSeek Harness 2/8] Produce a reproducible DeepSeek Harness candidate runtime**;
- #9331, **[DeepSeek Harness 3/8] Let users onboard and use the accepted Web and headless surfaces**;
- #9332, **[DeepSeek Harness 4/8] Preserve and recover accepted DeepSeek Harness state**;
- #9333, **[DeepSeek Harness 5/8] Run DeepSeek Harness through managed inference without provider credentials**;
- #9334, **[DeepSeek Harness 6/8] Enforce the accepted DeepSeek Harness trust and policy boundary**.

#9336 consumes the accepted receipt and owns public selection, supported inventory, documentation publication, and release-cohort activation.

## Problem Statement

A successful package install or local DSH prompt does not prove the supported NemoClaw product. The release candidate combines a particular NemoClaw commit, DSH package closure, architecture-specific managed image, startup profile, browser/headless surface, state contract, inference configuration, policy, OpenShell version, provider, and model.

DeepSeek Harness is prerelease software and its Web and headless modes exercise different process and session paths. Emulated image builds, source patches, manual sandbox edits, agent self-reporting, or evidence from a different digest cannot establish release support.

NemoClaw needs reproducible deterministic coverage plus fresh native live evidence on both accepted architectures while DeepSeek Harness remains a dark candidate.

## Desired Behavior

Repository tests prove the manifest, package, image, configuration, lifecycle, state, inference, browser, and security contracts deterministically.

One fresh native AMD64 environment and one fresh native ARM64 environment then install the exact candidate through the supported path, onboard the gated candidate without source mutation, exercise every Gate 1 surface, and complete the same versioned agent task through managed inference.

An independent host-side oracle evaluates workspace results, route receipts, process state, policy denials, persistent state, cleanup, and exact identities. DSH's final response is supporting evidence, not the success oracle.

## Qualification contract

Before the live run, commit a versioned qualification plan and evidence schema. The plan must freeze:

- the exact NemoClaw commit;
- DSH package version, integrity, lockfile digest, and native dependency closure;
- OCI index digest and the AMD64 and ARM64 image digests;
- manifest, startup-profile, generated-configuration, state-manifest, trust-matrix, and policy digests;
- OpenShell and accepted compute-runtime versions;
- accepted Web/headless surface and browser boundary;
- inference adapter, API family, provider, model, route identity, and capability settings;
- deterministic task version, fixtures, oracle version, timeouts, and retry policy;
- expected positive, negative, persistence, recovery, and cleanup assertions.

Any changed identity requires a new receipt. A rerun must produce a separate complete attempt record; it must not overwrite or conceal a failed attempt.

## Versioned agent task

Create a repository-owned `deepseek-harness-agent-v1` fixture with an `input.json` containing a run-unique nonce and the integer list `[17, 25, 58]`.

Through each accepted DSH surface, instruct the agent to:

1. read the fixture;
2. execute `uname -m` and record the observed architecture;
3. use its supported filesystem tools to write `result.json` containing the unchanged nonce, `sum: 100`, and observed architecture;
4. return a final response only after the file exists.

The host-side oracle must verify the exact JSON schema and values, the nonce, the native expected architecture (`x86_64` or `aarch64`), non-empty structured tool-call and tool-result correlation, a final response after tool completion, and the absence of raw unexecuted function-call text. It must reject a result derived only from assistant prose.

## Live Qualification Flow

For each architecture:

1. Provision a fresh supported host and record host, OpenShell, compute-runtime, and architecture identities.
2. Install the exact NemoClaw candidate through the supported install path.
3. Onboard the gated DeepSeek Harness candidate using only public or repository-supported candidate inputs.
4. Verify that onboarding selected the exact architecture digest and did not build a repository Dockerfile on the host.
5. Verify the recorded agent, package, image, startup profile, state manifest, generated configuration, inference route, model, and policy identities.
6. Exercise the Gate 3 browser boundary. For an accepted Web surface, prove authenticated forwarding or the exact accepted loopback-only limitation, and prove direct, unauthenticated, hostile Host/Origin, and untrusted forwarded-header access is denied as specified.
7. Run `deepseek-harness-agent-v1` through every accepted surface and evaluate it with the independent oracle and managed-route receipt.
8. Exercise an accepted multi-turn session, restart the runtime, and prove the accepted session and workspace state remain usable without restoring prohibited authority.
9. Snapshot or back up the accepted state, rebuild the sandbox from the same exact candidate, restore through the public recovery path, and repeat the oracle task.
10. Re-run representative Gate 6 denials for credential access, direct external egress, filesystem escape, host/container control, disabled plugins/features, and restored executable trust.
11. Change to another accepted managed route or model when Gate 1 includes that operation, rebuild, and prove no stale route remains.
12. Destroy the sandbox and verify the accepted cleanup contract: no live process, port forward, registered sandbox, retained prohibited state, or agent-owned lifecycle resource remains.
13. Collect bounded redacted evidence and verify it contains no provider credential, browser token, or prohibited state.

## Deterministic coverage

Repository-owned tests must cover:

- manifest schema, candidate-only registration, and empty accepted-receipt authority;
- exact package integrity, native install dependencies, compiled Web assets, OCI metadata, and multi-architecture image contracts;
- Web/headless dispatch, readiness, health, graceful shutdown, browser-boundary behavior, and command exit semantics;
- generated inference configuration, precedence, route changes, request compatibility, and credential canary scanning;
- state classification, concurrent-process behavior when accepted, restart, snapshot, backup, restore, rebuild, recovery, upgrade rejection, and destroy;
- trust-matrix enforcement and positive/negative filesystem, process, network, plugin, profile, browser, telemetry, and host-control behavior;
- candidate publication completeness and the atomic activation invariant without enabling DeepSeek Harness before Gate 8;
- compute-runtime and architecture neutrality in shared agent code.

## Required evidence and documentation

Store a repository-owned, machine-readable qualification manifest plus bounded human-readable summary. Each architecture receipt must include every frozen identity, all commands or workflow entry points, timestamps, attempt number, task/oracle results, route receipt, state and recovery results, security denials, cleanup result, CI links, and artifact locations.

The implementation must also add a maintainer qualification runbook explaining prerequisites, exact commands, artifact collection, redaction, retry handling, receipt review, and invalidation rules. Gate 8 must be able to link the accepted receipt without rewriting or hand-transcribing it.

## Constraints and Non-goals

- Do not qualify a floating package, mutable image tag, emulated live architecture, source patch, manual sandbox edit, or first-boot package installation.
- Do not substitute an image build for a native live AMD64 or ARM64 product run.
- Do not use an agent assertion as the only success oracle.
- Do not omit a failed attempt or merge evidence across different candidate identities.
- Do not expose credentials, browser tokens, route secrets, or prohibited state in logs or artifacts.
- Do not claim a surface, provider, API family, model, tool, plugin, platform, or lifecycle operation outside the Gate 1 matrix.
- Do not add DeepSeek Harness to public selection, the supported inventory, or the shipped managed-image cohort. Gate 8 owns the only activation change.
- Do not require unfinished Podman support when Gate 1 records a Docker-only launch.

## Acceptance Criteria

- [ ] Gates 1–6 are complete and their exact contracts are frozen in a versioned qualification plan.
- [ ] Deterministic tests cover every contract listed above and pass from the exact candidate commit.
- [ ] Package and publication tests bind one exact DSH closure to a complete AMD64/ARM64 OCI candidate cohort.
- [ ] Candidate-only tests prove no accepted qualification receipt or public selection is enabled before Gate 8.
- [ ] A fresh native AMD64 environment completes the entire live flow without manual repair.
- [ ] A fresh native ARM64 environment completes the entire live flow without manual repair.
- [ ] Every Gate 1 surface completes `deepseek-harness-agent-v1`, and the independent oracle verifies its file, nonce, sum, architecture, tool correlation, and final response.
- [ ] A managed-route receipt proves each qualifying model turn used the exact accepted adapter, API family, provider, and model.
- [ ] The accepted browser boundary passes positive and adversarial reachability tests when Web is in scope.
- [ ] Restart, snapshot/backup, rebuild, restore, recovery, route change when accepted, and destroy pass through public product paths.
- [ ] Representative credential, egress, filesystem, host-control, disabled-feature, and restored-trust denials pass on both architectures.
- [ ] Stock onboarding uses exact image digests and performs no host Dockerfile build.
- [ ] Cleanup leaves no live DSH process, forward, registered sandbox, or prohibited retained state.
- [ ] Machine-readable receipts include every required identity, attempt, result, link, and redaction assertion.
- [ ] A maintainer can reproduce the qualification from the committed runbook.
- [ ] The accepted evidence is secret-free and ready for Gate 8 to bind atomically.

## Category

Testing

## Checklist

- [x] I searched existing issues and this is not a duplicate.
- [x] I described the problem and desired behavior.

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or test paths. First confirm completion of #9329–#9334, then locate the qualification-plan, receipt-schema, fixture, oracle, deterministic-test, and maintainer-runbook entry points. Done means both native architectures complete the specified flow with complete secret-free receipts and reproducible evidence for Gate 8.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, typescript
Domain
devops, infrastructure, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.