[PIR][WP31] Repo-native optimization manifest and disciplined experiment loop (ADR-335)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 4.5k
- Forks
- 603
- Avg merge
- 23h 32m
- Merged PRs (30d)
- 59
Description
Part of the PIR program epic #837. See 12-wave5-evidence-review.md for evidence grades and 13-wave5-program-plan.md for the WP/ADR mapping. ADRs land via #911.
This is an extension, not a build
An at-source audit of origin/main found roughly 80% of the priority-1 mechanism already implemented in crates/ruvector-sota-bench/harness/:
| Element | Status | Where |
|---|---|---|
| Objective function | Exists, hard-coded | metrics.ts darwinScore() → flywheel.ts toScore() |
| Benchmark command | Exists, closed + pinned | benchmark.ts runObservedBenchmark — isolated binary, env fingerprint, gitCommit(), RSS sampling, cache key |
| Protected invariants | Exists, right semantics | vetoes.ts composeVetoProviders — "any one provider can block promotion; none can rescue it." Seven providers shipped |
| One isolated variable | Exists, enforced twice | flywheel.ts nextParameter steps one lever one notch; benchmark.ts normalizePolicy() rejects unknown levers |
| Cost budget | Exists, three places | costPerWin; resource_cost_worsened above 1.05×; darwin.ts budget/ceiling/risk |
| Promotion rules | Exists | ruvectorPromotionRule + gateFingerprint() + verifyReplayBundle() + disjoint holdout/anchor assertion |
| Keep / discard / reject | Exists as three-way | statistics.ts pairedBootstrapDecision |
The two real gaps
- No per-repo declaration layer. Everything above is hard-coded in TypeScript or declared per-experiment. A second repo adopting this loop today would have to fork the harness. → new
schemas/optimization-manifest-v1.json, from which each experiment'sresearch-manifest-v1.jsonis derived. - The Pareto frontier is not ours.
darwin.tspassesselection: "pareto"to the external@metaharness/darwin@^0.9.1. Decide explicitly: own an in-repo frontier, or accept the dependency with a recorded rationale. (environmentDesigner.ts'satFrontieris a different frontier — task difficulty, not Pareto.)
Blocking security requirement
Intersect, never substitute. A manifest is an untrusted proposal (Invariant 7). It may narrow the lever set, tighten a budget, or add invariants. It may never widen benchmark.ts's closed lever set, lower minimumEffect, remove a veto provider, or relax the disjoint holdout/anchor requirement — rejected at load with a typed error, not silently clamped. A config file that can weaken a promotion gate is a privilege-escalation primitive wearing a config file's clothes.
Evidence caveat
NVIDIA's 15–77% throughput range is n=2 ("one Claude Code pair and one Codex pair"). It is a directional anecdote, not a success target. The bar is this program's own recomputed research-gate delta. Cite the PR body's stronger evidence instead: every produced recipe was independently re-deployed and re-benchmarked on SGLang/GB200 and vLLM/H100, "reproducing or beating the agents' claimed numbers."
Acceptance
schemas/optimization-manifest-v1.jsonexists and validates offline (hash-locked$refresolution, perscripts/research-gate/)- Derivation populates
research-manifest-v1.jsonwithout changing its required fields - Regression tests: a manifest attempting each prohibited widening is rejected
- Pareto decision recorded in the ADR
- Harness
npm run check && npm test+scripts/research-gate/tests pass
Repo: ruvnet/ruvector only.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/research/perpetual-intelligence-runtime/12-wave5-evidence-review.md and 13-wave5-program-plan.md, then inspect the existing harness files under crates/ruvector-sota-bench/harness/ and the scripts/research-gate/ tests. Record the Pareto decision in the ADR, add the hash-locked manifest schema and derivation, and verify that every prohibited widening is rejected while npm run check, npm test, and research-gate tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json, rust, typescript
- Domain
- ai, security, testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100