Qualify evidence report v2 for production source authenticity and provider budgets

Open
#3 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
github-actions, python, rust

Research direction

Begin by reviewing merged PR4 and PR5 and the exact Reports CI and MetaHarness CI runs named in the issue; no implementation files or test paths are identified. Map the five open production gates to existing 18-test and CI coverage, then define authenticated retained evidence, zero-effect failure behavior, tenant isolation and retention, outage and restart tests, and any required ADR or quality evaluation as completion criteria.

Written by the indexing model from the issue text.

Description

The bounded v2 evidence report revival is merged in PR4, merge commit 4702b5124aa3740e2d868f62dad86caef6fd00b8.

Exact tested PR head: 883d9587d55cfcf7a6c446c802faeee42da9e969. Local and remote tree matched a5e4db1c1a5234318fa0c8f75097b25b5916adbf.

Validation:

  • Reports CI: success, 17 domain/provider/CLI/MCP tests and release artifacts.
  • MetaHarness and Autogenous CI: success, generated profile tests/build/doctor/audit/provenance/Darwin plus Rust hard gates.
  • Independent review reran 17 tests and deliberately caused real domain test failure to confirm MCP returns both isError:true and passed:false.
  • npm audit reported zero root vulnerabilities. Fixture benchmark p95 2.67 ms for 100 sources over 100 iterations; no SOTA or semantic quality claim.

Implemented: pinned Agentic Search BM25, source snapshots/hashes, exact citation verification, freshness filtering, report comparison, escaped HTML, six official SDK MCP tools and CLI, generated MetaHarness profiles and Autogenous gates. Historical unauthenticated paid network launch paths are retired. Optional fixed Exa CLI discovery is explicitly gated and absent from MCP.

Production gates remain open:

  1. Establish trusted source acquisition, capture timestamp provenance and a signing identity. Local hashes currently prove consistency only.
  2. Validate Exa with an operator-owned account and spending cap. Controlled protocol fixtures do not qualify live service behavior.
  3. Define private tenant OS/process isolation, report storage access controls and retention. Reports deliberately contain source text.
  4. Test real provider outages and restart behavior before automating customer workflows.
  5. Future LLM synthesis or RuVector semantic ranking needs held out citation/quality evaluation, cost/latency comparison and a separate ADR.

Acceptance: production reports trace each citation to authenticated retained evidence; tampered reports and unauthorized provider requests cause zero external effects. Local and exact committed CI tests remain passing. No automatic federation publication or promotion is enabled. The historical PyPI distribution was not updated by this Git merge.

Optimization followup: PR5 merged as 1459a17882e9f2ae98833d703df91c425c675cb8 after independent review and exact head d5304b5731bd264db7f111f69d346b9328b94f9f passed both Reports CI and MetaHarness CI.

Verification no longer performs two redundant canonical serialization passes after fully regenerating the expected report. No source cache or validation shortcut was introduced. All 18 tests pass; 40 varied baseline/candidate cases preserve complete report and citation results. Controlled median of seven repetition p95 results: 2.788 to 2.270 ms for one 100 source report, and 24.469 to 18.427 ms for ten independent reports, reductions of 18.6% and 24.7%. Existing bounds and production gates remain unchanged. Raw measurements and the reproducible fixed baseline script are committed; these are local verification gains, not SOTA research quality claims.

Dominant language
Python
Stars
53
Forks
8
Avg merge
2m
Merged PRs (30d)
2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruvnet/agentic-reports

All issues in ruvnet/agentic-reports

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.