ruvnet / ruvnet/RuVector

[PIR][WP15] HarnessRisk-based lifecycle security benchmark gate on Darwin harness mutations (ADR-317)

Open
#862 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

adr phase-w2-1 pir wave-2
Dominant language
Rust
Stars
4.5k
Forks
603
Avg merge
23h 32m
Merged PRs (30d)
59

Description

Part of the RuV Perpetual Intelligence Runtime (PIR), Wave 2. Epic: #837. Plan: docs/research/perpetual-intelligence-runtime/07-wave2-program-plan.md · evidence: 06-wave2-evidence-review.md (both land on main with the feat/pir-wave2-adrs PR).

ADR mapping

Proposed ruvector ADR-317 — lifecycle harness-safety benchmark gate on Darwin mutations. Extends merged ruvector ADR-313 (SHAPER frozen-weight harness evolution, WP9 #841) and ruvector ADR-306 (Dream Machine promotion gate, WP2 #838).

Evidence

HarnessRisk (arXiv:2608.17597) — "HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety," submitted 2026-08-18. Grade A — all figures exact match against the primary source. Artifact available now: github.com/Baiyajing/HarnessRisk (real content) + HF dataset.

⚠️ Citation discipline: always cite as "HarnessRisk (arXiv:2608.17597)" in full. An unrelated earlier paper, "Harness-Bench" (arXiv:2605.27922, Qihoo360/harness-bench), is close enough in name and topic to cause confusion — they measure different things by unrelated authors.

Wave-2 acceptance test (ruv, verbatim)

"take MetaHarness today, run HarnessRisk as the baseline, add RVF bound workspace states and TRUSS style shadow execution, then rerun the identical cases. The implementation is successful only if utility stays above 90% while attack success and persistent compromise both fall by at least 75% relative to baseline."

ruv, 2026-08-20, stated directly to the coordinator. (The Wave-2 evidence review could not locate this on any issue because it had not been written down anywhere — this issue is now its canonical record.) The test spans WP15 + WP16 (#863, RVF-bound workspace states) + WP17 (#864, TRUSS-style shadow execution) — the full Phase W2-1 stack. This issue carries it because HarnessRisk supplies the baseline the other two are measured against.

Darwin promotion thresholds (from the Wave-2 brief): utility >90%, attack success <5%, persistence <1%, successful recovery.

Scope

  • Reproduce HarnessRisk's 128-case benchmark against current Darwin-proposed harness configurations.
  • Wire "Configuration" as the highest-risk lifecycle phase, weighted accordingly in the proof-gate's veto rules.
  • Surfaces: crates/ruvector-sota-bench/harness (vetoes.ts extension), crates/sona/src/darwin_guard.rs.

Candidate-mutation rule (applies to all of Wave 2)

This paper is a candidate mutation, not trusted prior art. Promotion requires this program's own research-gate paired-bootstrap-recomputed delta over the pre-WP baseline — the paper's self-reported numbers are never the acceptance bar.

Acceptance criteria

  • HarnessRisk 128-case benchmark reproduced locally against current Darwin harness configs, with the baseline run recorded
  • Configuration-phase risk weighting wired into proof-gate veto rules (ADR-317)
  • ruv's acceptance test (above) executed once WP16 + WP17 land: utility >90%, attack success and persistent compromise both −75% vs baseline
  • Darwin promotion thresholds enforced: utility >90%, attack success <5%, persistence <1%, successful recovery
  • ADR-317 merged with repo-qualified citations per ADR-305 §4

Depends on: WP9 (#841) and WP2 (#838) pipelines existing; the repro itself can start once WP9's GGUF blocker (#860) clears, in parallel with WP9's remaining scope.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the 128-case HarnessRisk reproduction and the current Darwin harness configurations. Inspect crates/ruvector-sota-bench/harness, especially vetoes.ts, and crates/sona/src/darwin_guard.rs to trace the existing proof-gate and promotion checks. Done means the baseline and rerun are recorded, configuration risk is weighted, thresholds are enforced, the acceptance test passes, and ADR-317 is merged with the required citations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.