[PIR][WP21] SPADE-pattern self-play environment generation + REQUIRED external-grounding veto (ADR-324)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 4.5k
- Forks
- 603
- Avg merge
- 23h 32m
- Merged PRs (30d)
- 59
Description
Epic: #837 · Wave 3, Phase W3-1 (top priority) · Proposed ADR-324 · Paper: SPADE — Self-Play in Adaptive Synthetic Executable Environments, arXiv:2608.19197 (Grade A, all figures exact-match; reference implementation live at github.com/spade-rl/spade, MIT, real content)
⚠️ CITATION DISCIPLINE — NEVER write bare "SPADE". Always cite as SPADE (arXiv:2608.19197, Self-Play in Adaptive Synthetic Executable Environments) on first use in every code comment, ADR, commit message, PR, and status update. "SPADE" alone collides — in the same multi-agent domain — with the long-established SPADE (Smart Python multi-Agent Development Environment) framework. This is the most severe name collision of any PIR wave; a bare reference reads as a completely reasonable citation to the wrong, decades-old project. (
08-wave3-evidence-review.md§1)
Goal
Add an Environment-Designer mutation-candidate source to Darwin's proposal surface: an LLM-authored, executable Gym-style (reset() / step()) training environment, generated by the same frozen model under a regret signal, as a new type of Darwin mutation candidate — the same "one more candidate-generation strategy feeding the existing pipeline" shape as WP19 SkillForge (ADR-321). Adapt the released spade-rl/spade implementation rather than rebuilding from the paper.
Extends merged ADR-306 (dream-machine / research-gate promotion gate), ADR-313 (SHAPER frozen-weight loop, WP9), ADR-321 (SkillForge candidate-source pattern, WP19).
🔒 REQUIRED acceptance criterion — external-grounding veto (the load-bearing item, ahead of any feature work)
Every generated environment MUST trace to external evidence — repository tests, real observed/incident failures, physical-world observations, or an independently published benchmark. A self-invented closed loop where the same system that generated the environment is also the sole judge of its value is a REJECT. This is a hard acceptance criterion, not a nice-to-have:
- Implement a new hard veto in
research-gate(crates/ruvector-sota-bench/harness/src/statistics.ts, alongside the existing paired-bootstrap statistical veto) requiring every generated environment to carry a traceable external-grounding link before it can reach the promotion gate at all. - The veto MUST fire before Dream Machine's evaluation stage ever runs, not after.
- Apply ADR-315's constitutional gate to any generated environment used to certify a promotion, not only to environments used for training — a subtly-wrong environment corrupts the measurement of "better", poisoning every downstream promotion that trusts it. This is strictly more capability-sensitive than WP19's synthetic issues or WP17's shadow-executed skills.
Rationale (evidenced, not hypothetical): SPADE's own ablation found external corpus grounding "critical to success" — the paper's authors themselves observed the ungrounded variant underperforming. Program invariant 5 ("every promoted mutation must outperform its parent") is not sufficient to catch this: a mutation can outperform its parent on a self-generated benchmark and still be worthless against reality.
Wave-3 acceptance test (ruv, verbatim 2026-08-20)
"Freeze the underlying model for 30 days; allow only harness, memory, topology, tools, and generated environments to evolve; hold out 20% of real tasks the evolution system never sees; if held-out success improves at least 10% while capability violations remain zero and every promoted change has reproducible provenance, you have demonstrated genuine substrate-level learning rather than benchmark adaptation."
Composes with the existing WP12 30-day acceptance harness + ADR-315's capability-expansion gate; adds (a) a genuinely held-out 20% task split the evolution system never trains or self-evaluates against, and (b) a ≥10% held-out-success threshold measured the same paired-bootstrap way research-gate measures everything else.
Preprint-reproduction rule
The paper's numbers are hypotheses, not the acceptance bar. A live reference repo changes how much code must be written from scratch — it does not change what counts as evidence. No promotion without an independently recomputed research-gate paired-bootstrap delta over the pre-WP baseline.
Scope / files
New environmentDesigner.ts alongside the existing candidate-source files in crates/ruvector-sota-bench/harness/src/ (darwin.ts, dreamMachine.ts, shaperLoop.ts, skillForge.ts), plus the new veto function in statistics.ts's hard-veto set.
Depends on
WP9 (#841, SHAPER loop) · WP2 (#838, dream-machine / research-gate promotion) · WP19 (#866, SkillForge candidate-source pattern to mirror)
Gates
Run npx @claude-flow/cli@latest security scan after landing (touches a mutation-proposal surface). Highest-security-sensitivity item in this wave.
Sources: 08-wave3-evidence-review.md (evidence grade + artifact status), 09-wave3-program-plan.md (WP/ADR mapping). Filed by the issue steward after coordinator approval of the Wave-3 breakdown.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read skillForge.ts, darwin.ts, dreamMachine.ts, shaperLoop.ts, and the hard-veto set in statistics.ts. Trace how candidate sources enter research-gate, then define the external-grounding check before Dream Machine evaluation. Done requires the held-out 20% task split, paired-bootstrap threshold, reproducible provenance, zero capability violations, and the required security scan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, typescript
- Domain
- ai, security, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100