ruvnet / ruvnet/RuVector

[PIR][WP21] SPADE-pattern self-play environment generation + REQUIRED external-grounding veto (ADR-324)

Open
#880 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

adr phase-w3-1 pir security wave-3
Dominant language
Rust
Stars
4.5k
Forks
603
Avg merge
23h 32m
Merged PRs (30d)
59

Description

Epic: #837 · Wave 3, Phase W3-1 (top priority) · Proposed ADR-324 · Paper: SPADE — Self-Play in Adaptive Synthetic Executable Environments, arXiv:2608.19197 (Grade A, all figures exact-match; reference implementation live at github.com/spade-rl/spade, MIT, real content)

⚠️ CITATION DISCIPLINE — NEVER write bare "SPADE". Always cite as SPADE (arXiv:2608.19197, Self-Play in Adaptive Synthetic Executable Environments) on first use in every code comment, ADR, commit message, PR, and status update. "SPADE" alone collides — in the same multi-agent domain — with the long-established SPADE (Smart Python multi-Agent Development Environment) framework. This is the most severe name collision of any PIR wave; a bare reference reads as a completely reasonable citation to the wrong, decades-old project. (08-wave3-evidence-review.md §1)

Goal

Add an Environment-Designer mutation-candidate source to Darwin's proposal surface: an LLM-authored, executable Gym-style (reset() / step()) training environment, generated by the same frozen model under a regret signal, as a new type of Darwin mutation candidate — the same "one more candidate-generation strategy feeding the existing pipeline" shape as WP19 SkillForge (ADR-321). Adapt the released spade-rl/spade implementation rather than rebuilding from the paper.

Extends merged ADR-306 (dream-machine / research-gate promotion gate), ADR-313 (SHAPER frozen-weight loop, WP9), ADR-321 (SkillForge candidate-source pattern, WP19).

🔒 REQUIRED acceptance criterion — external-grounding veto (the load-bearing item, ahead of any feature work)

Every generated environment MUST trace to external evidence — repository tests, real observed/incident failures, physical-world observations, or an independently published benchmark. A self-invented closed loop where the same system that generated the environment is also the sole judge of its value is a REJECT. This is a hard acceptance criterion, not a nice-to-have:

  • Implement a new hard veto in research-gate (crates/ruvector-sota-bench/harness/src/statistics.ts, alongside the existing paired-bootstrap statistical veto) requiring every generated environment to carry a traceable external-grounding link before it can reach the promotion gate at all.
  • The veto MUST fire before Dream Machine's evaluation stage ever runs, not after.
  • Apply ADR-315's constitutional gate to any generated environment used to certify a promotion, not only to environments used for training — a subtly-wrong environment corrupts the measurement of "better", poisoning every downstream promotion that trusts it. This is strictly more capability-sensitive than WP19's synthetic issues or WP17's shadow-executed skills.

Rationale (evidenced, not hypothetical): SPADE's own ablation found external corpus grounding "critical to success" — the paper's authors themselves observed the ungrounded variant underperforming. Program invariant 5 ("every promoted mutation must outperform its parent") is not sufficient to catch this: a mutation can outperform its parent on a self-generated benchmark and still be worthless against reality.

Wave-3 acceptance test (ruv, verbatim 2026-08-20)

"Freeze the underlying model for 30 days; allow only harness, memory, topology, tools, and generated environments to evolve; hold out 20% of real tasks the evolution system never sees; if held-out success improves at least 10% while capability violations remain zero and every promoted change has reproducible provenance, you have demonstrated genuine substrate-level learning rather than benchmark adaptation."

Composes with the existing WP12 30-day acceptance harness + ADR-315's capability-expansion gate; adds (a) a genuinely held-out 20% task split the evolution system never trains or self-evaluates against, and (b) a ≥10% held-out-success threshold measured the same paired-bootstrap way research-gate measures everything else.

Preprint-reproduction rule

The paper's numbers are hypotheses, not the acceptance bar. A live reference repo changes how much code must be written from scratch — it does not change what counts as evidence. No promotion without an independently recomputed research-gate paired-bootstrap delta over the pre-WP baseline.

Scope / files

New environmentDesigner.ts alongside the existing candidate-source files in crates/ruvector-sota-bench/harness/src/ (darwin.ts, dreamMachine.ts, shaperLoop.ts, skillForge.ts), plus the new veto function in statistics.ts's hard-veto set.

Depends on

WP9 (#841, SHAPER loop) · WP2 (#838, dream-machine / research-gate promotion) · WP19 (#866, SkillForge candidate-source pattern to mirror)

Gates

Run npx @claude-flow/cli@latest security scan after landing (touches a mutation-proposal surface). Highest-security-sensitivity item in this wave.


Sources: 08-wave3-evidence-review.md (evidence grade + artifact status), 09-wave3-program-plan.md (WP/ADR mapping). Filed by the issue steward after coordinator approval of the Wave-3 breakdown.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read skillForge.ts, darwin.ts, dreamMachine.ts, shaperLoop.ts, and the hard-veto set in statistics.ts. Trace how candidate sources enter research-gate, then define the external-grounding check before Dream Machine evaluation. Done requires the held-out 20% task split, paired-bootstrap threshold, reproducible provenance, zero capability violations, and the required security scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, typescript
Domain
ai, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.