[Blocked until D1 evidence] D2/D3 comparative falsification vs native tool search

Summary

Build the decisive competitive evaluation for ContextWeaver's product thesis.

The old framing — "naive all tools/full history" versus ContextWeaver — is retained only as a historical sanity baseline. It is not sufficient evidence for adoption because modern providers and runtimes increasingly offer native tool search/deferred loading and programmatic tool-use mechanisms.

The benchmark must answer:

Does ContextWeaver add meaningful value beyond strong contemporary alternatives, and if so where?

This is a product gate, not a marketing benchmark. Losses must be published as clearly as wins.

Hypotheses under test

H2 — value beyond native tool search

ContextWeaver or ContextWeaver + native tool search should provide a meaningful advantage in at least one non-trivial dimension — task success, capability selection/argument correctness, ambiguity handling, reproducibility/drift control or comparable operational evidence — without materially degrading outcome quality.

H3 — independent value from artifact-backed result/context handling

The ContextWeaver firewall/artifact path should be evaluated against contemporary provider/runtime mechanisms that keep intermediate tool data out of model context. The relevant possible advantages are containment, provenance, deterministic replay, sensitivity handling, portability and reproducibility — not merely character/token reduction.

Required benchmark arms

Use the same held-out tasks/catalogs wherever the mechanism permits.

1. Naive all-tools / full-result baseline

   • historical sanity/control arm only;
   • never the sole comparison behind a public claim.

2. Provider-native tool search / deferred loading

   • use supported current native mechanisms for at least one major provider;
   • record exact model/API feature/date and configuration.

3. Simple client-side retrieval baseline

   • deliberately cheap, understandable retrieval over tool descriptions (for example BM25/TF-IDF or a straightforward semantic top-k baseline);
   • prevents the benchmark from attributing generic retrieval value to ContextWeaver architecture.

4. ContextWeaver routing only

   • isolate the incremental value/cost of the routing layer.

5. ContextWeaver full compiled/runtime path

   • use compiled/evaluated bundle plus bounded route/hydrate/context behavior where available.

6. ContextWeaver + provider-native tool search

   • test complementarity instead of assuming replacement;
   • ContextWeaver may bound/validate the candidate surface while provider-native mechanisms perform late binding.

If a provider-native feature cannot be reproduced through an OpenAI-compatible endpoint, use a provider-specific adapter in the benchmark harness rather than omitting the strongest baseline.

Result-handling / firewall ablation

Run a separate ablation on representative large/intermediate tool outputs:

• raw result directly returned to model;
• provider/runtime-native programmatic/filtering mechanism where available;
• ContextWeaver artifact-backed firewall/result path;
• ContextWeaver + native mechanism where meaningful.

Measure not only prompt size but task outcome and properties such as:

• whether the model received unnecessary raw sensitive/noisy material;
• ability to inspect/replay what was retained versus externalized;
• provenance/handle integrity;
• deterministic reproducibility;
• latency and cost.

Datasets and scenario design

Held-out discipline

• Separate benchmark design from tuning.
• Do not optimize ContextWeaver against the final evaluation set.
• Version every dataset and catalog.
• Record authorship/provenance for scenarios.

Required scenario mix

Use at least:

• existing committed scenarios for continuity with historical numbers;
• the killer-demo heterogeneous catalog;
• larger distractor catalogs;
• ambiguous/cross-namespace cases from #492;
• argument/schema correctness cases;
• large tool-result workflows;
• at least some externally sourced or adopter-derived capability catalogs/workflows that were not invented solely to demonstrate ContextWeaver.

Sanitize adopter-derived fixtures before committing them.

Metrics

Outcome quality

• end-to-end task success;
• correct capability/tool selected;
• schema/argument correctness;
• false exclusion / correct tool missing from candidate set;
• ambiguity detection / correct clarification behavior;
• answer correctness where deterministically checkable.

Prefer deterministic task-specific checkers. Use an LLM judge only when unavoidable, commit the rubric/prompt, and label the metric clearly.

Efficiency

• prompt/input tokens;
• output tokens where relevant;
• latency p50/p95;
• provider cost estimate / actual usage cost where available;
• routing/compile overhead separately from model latency.

Reproducibility / control evidence

For the compiled path, report where applicable:

• bundle identity/version;
• source coverage and provenance;
• drift/diff detected between candidate and last-good surface;
• deterministic replay success;
• provider/host portability of the same bundle/surface.

Do not collapse these into a fake single score. They are distinct value dimensions.

Statistical discipline

• temperature 0 / deterministic settings where supported;
• multiple trials where model behavior remains stochastic (target n≥5 for headline claims; n≥3 minimum for expensive exploratory runs);
• confidence intervals/variance for headline rates;
• fixed seeds where supported;
• report exact model identifiers, API dates, region if relevant and run date;
• record failures/timeouts rather than silently retrying them away.

Anti-cherry-picking rules

• Commit benchmark definitions before publishing final results where practical.
• Keep failed/negative scenarios in the report.
• Do not drop a provider-native arm because it wins.
• Do not tune only the ContextWeaver arm with bespoke prompts unavailable to baselines; any arm-specific instructions must be documented and justified.
• Public README claims must point to a versioned report and exact configuration.

Harness architecture

Create/extend benchmarks/e2e/ with explicit strategy adapters, for example:

BenchmarkStrategy
├── NaiveStrategy
├── SimpleRetrievalStrategy
├── ProviderNativeToolSearchStrategy
├── ContextWeaverRoutingStrategy
├── ContextWeaverCompiledStrategy
└── ContextWeaverPlusNativeStrategy

Keep model-provider adapters separate from benchmark strategies so the same strategy can be exercised across providers where supported.

Environment configuration should support:

• provider/model identifier;
• API base where applicable;
• credentials via environment only;
• fixed budget cap (--budget-usd);
• selected strategy/arm;
• trial count;
• dataset/catalog version.

Never log API keys or adopter secrets.

Outputs

Every run emits:

• machine-readable, versioned JSON with raw per-task/per-trial results;
• rendered comparative Markdown report;
• environment/config metadata excluding secrets;
• aggregate + per-scenario/per-case-kind tables;
• an explicit where ContextWeaver lost section;
• limitations/threats-to-validity section.

Publish a stable docs/e2e_report.md or equivalent generated report for the latest approved run and retain historical reports/results by version/date.

Decision gates

The benchmark should inform #758 rather than automatically justify the current architecture.

Go evidence

Examples of acceptable evidence:

• measurable outcome-quality advantage over native/simple baselines;
• similar quality with materially better reproducibility/control/provenance that target users value;
• ContextWeaver + native provider features outperform either alone in important workflows;
• artifact-backed result handling materially improves containment/replay/provenance without unacceptable latency/quality cost.

Negative evidence

Treat these as reasons to shrink/reposition:

• ContextWeaver only beats the naive baseline;
• native tool search is consistently as good or better and ContextWeaver adds meaningful latency/complexity;
• the full compiler/runtime path produces no adopter-relevant benefit over routing-only/simple retrieval;
• artifact-backed result handling has no useful advantage over native mechanisms;
• gains exist only on synthetic fixtures engineered around ContextWeaver.

Acceptance criteria

• At least the six routing/tool-selection arms above are implemented or an explicit evidence-backed reason is recorded for any unavailable arm.
• At least one current provider-native tool-search/deferred-loading mechanism is exercised directly.
• Simple retrieval baseline is included.
• Routing-only and full-compiled ContextWeaver arms are separated.
• ContextWeaver + native complementarity is measured.
• Result-handling/firewall ablation includes a contemporary native/runtime comparison where available.
• Dataset contains held-out ambiguous, argument-correctness and large-result cases plus at least some external/adopter-derived scenarios.
• Report covers outcome quality, false exclusion, ambiguity, tokens, latency and cost.
• Reproducibility/drift evidence is reported separately for compiled paths.
• n≥3 trials for stochastic headline comparisons; target n≥5 for launch claims.
• Raw versioned JSON + rendered report committed/published.
• Negative results and limitations are visible.
• Public product claims are not updated until this comparative evidence exists.

Out of scope

• public leaderboard infrastructure;
• optimizing specifically for leaderboard scores;
• gating ordinary PR CI on paid/network model calls;
• fine-tuning providers/models;
• pretending one benchmark proves universal superiority.

Related

• #758 — product thesis and falsification gates;
• #492 — expanded routing gold set/ambiguity cases;
• #489 — adversarial evaluation;
• #434 — killer demo/tutorial;
• #433 — neutral competitive landscape;
• #397 — evidence-based claims;
• #561 — host-provided routing/task outcomes.

---

Priority recommendation: P0 during the validation programme. Broad distribution should not depend on weaker naive-only claims.

Start in benchmarks/e2e/ and inspect the existing benchmark harness before defining the strategy adapters and provider-specific adapters described here. Implement the required comparison and result-handling arms, versioned datasets and raw JSON outputs, then generate docs/e2e_report.md with per-scenario results, negative findings, limitations, and the specified outcome, efficiency, and reproducibility metrics.