NVIDIA / NVIDIA/NemoClaw

Run speculative compatibility tests against candidate OpenShell, OpenClaw, Hermes, and DCode versions

Open
#6,691 0 comments 0 reactions 1 assignee Claimed by @cv View on GitHub
area: ci area: e2e integration: hermes integration: openclaw v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Goal

Provide a maintainer-triggered compatibility path that runs NemoClaw's existing deterministic and live tests against speculative candidate versions of OpenShell, OpenClaw, Hermes, and DCode. The path should produce exact, auditable compatibility evidence without requiring committed pin edits or a generated pull request.

This issue supersedes the remaining dependency-pin canary scope from #5242, but narrows the desired output to test evidence rather than automated pin changes. PR #6619 delivered dependency-pin drift detection and established that the operational installer, blueprint, Dockerfile, and agent files remain authoritative.

The first production use is #6726: validate that PR's exact head against OpenShell v0.0.82 when immutable release artifacts exist. #6726 remains the OpenShell upgrade PR; this issue must not replace it or create a competing PR.

## Current state

- `scripts/checks/dependency-pins.ts` derives committed pins from their operational authorities and rejects drift.
- Normal PR CI can validate a NemoClaw commit, and `.github/workflows/e2e.yaml` declares live E2E coverage against an exact SHA.
- Existing install and test paths generally consume the committed dependency versions, so trying another version requires source edits or a purpose-built PR before meaningful CI and E2E evidence can run.
- Component-specific update helpers exist, but changing pins is not the hard part and is not the focus of this issue.
- Nothing currently accepts an exact NemoClaw ref plus a dependency candidate, injects that candidate only into the test environment, and reports which deterministic and live tests passed against the resolved runtime.

## Scope

- Add a manually dispatchable candidate-compatibility entry point for the initial supported components: `openshell`, `openclaw`, `hermes`, and `dcode`.
- Accept a NemoClaw ref and candidate version or tag. Resolve the NemoClaw ref once to a full commit SHA and bind every job and result to that SHA.
- Resolve and verify immutable candidate metadata from the component's official published source, including release checksums, npm integrity and tarball URLs, or release-asset SHA-256 as applicable.
- Inject the candidate only at the install and test boundary. Do not edit committed pin authorities, create a branch, push a commit, or open a pull request.
- Run the normal deterministic test lanes against the candidate environment, including source/unit, integration, installer, package-contract, plugin, and E2E-support coverage where applicable.
- Reuse `.github/workflows/e2e.yaml` as the live E2E source of truth. Run the component-appropriate live tests at the same NemoClaw SHA and record every selected, skipped, and not-applicable test.
- Prove the candidate actually reached each exercised runtime by recording the observed installed or reported version before accepting test results.
- Publish a workflow summary and durable artifact containing the NemoClaw SHA, component, requested candidate, resolved artifact identities and digests, observed runtime versions, test results, logs, rerun identity, and skipped-test reasons.
- Make reruns for the same NemoClaw SHA, component, and candidate idempotent and auditable.
- Keep candidate execution fail-closed around credentials. Initial support is limited to official published versions or tags; executing arbitrary upstream PR artifacts with E2E secrets requires a separate provenance and approval design.

## Acceptance criteria

- [ ] A maintainer can dispatch a compatibility run for an exact NemoClaw ref and a candidate OpenShell, OpenClaw, Hermes, or DCode version/tag.
- [ ] The run resolves the NemoClaw ref to one full SHA and records immutable candidate artifact identities and digests before executing candidate code.
- [ ] Candidate overrides are ephemeral: the run produces no committed diff, branch, pull request, or pin-authority update.
- [ ] Normal deterministic test coverage runs against the candidate environment, and failures identify the exact lane, candidate, and NemoClaw SHA.
- [ ] Component-appropriate live E2E coverage declared by `.github/workflows/e2e.yaml` runs against that same SHA, with explicit evidence for every selected or skipped test.
- [ ] Every candidate-aware job verifies and reports the runtime version it actually exercised; a version mismatch fails the run.
- [ ] The evidence artifact is sufficient to reproduce the run and compare results across candidates without relying on mutable tags or workflow UI state.
- [ ] A failed or cancelled run does not move `latest` or `lkg`, publish production images or releases, merge code, or update external public surfaces.
- [ ] Tests cover invalid input, upstream metadata failure, digest or integrity mismatch, candidate propagation, observed-version mismatch, reruns, and the credential/provenance boundary.
- [ ] The first production use validates #6726's exact head against the published OpenShell v0.0.82 artifacts and records the relevant deterministic and live E2E evidence.

## Non-goals

- Automatically updating committed dependency pins or generating pin-bump pull requests.
- Centralizing pins in YAML or introducing a committed candidate manifest or generated mirror.
- Executing untrusted or arbitrary upstream PR artifacts with NemoClaw E2E secrets.
- Moving or redefining `latest` or `lkg`.
- Publishing production images or releases.
- Updating Brev, Spark, public documentation, or other external consumers.
- Treating a candidate run as release readiness or stateful upgrade proof when the corresponding live test was not selected and executed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.