[RFC / Research Plan] Benchmarking Protocol Adherence, Model Performance, and Citation Fidelity across AI Coding Agents

$(cat << 'EOF'

[RFC / Research Plan] Benchmarking Protocol Adherence, Model Performance, and Citation Fidelity across AI Coding Agents

Summary

We propose a comprehensive benchmarking framework and study to evaluate how AI coding agents execute standardized computational biology protocols. Specifically, this plan investigates:

1. Model Generalizability: How protocol execution varies across capability tiers (frontier, lightweight, open-weight) rather than specific hardcoded models.
2. Behavioral & Performance Delta: How the presence of a structured protocol.md alters agent behavior, accuracy, and efficiency compared to unguided baselines.
3. Citation & Provenance Fidelity: Whether agents faithfully report primary literature citations (DOIs/PMIDs) from atomic (1 citation) and composite (1+ aggregated citations) protocols, and how an invoking agent skill can reinforce this behavior.

---

Motivation & Core Questions

As we develop AI-agent-executable protocols (ADR 0001, ADR 0004), we must systematically measure how effectively agents consume these specifications and whether they maintain scientific rigor during execution.

1. Performance Across Capability Tiers

• Question: Do lightweight or cost-effective models follow protocols as effectively as flagship reasoning models?
• Implication: Understanding model tier requirements helps users balance computational cost with execution reliability when dispatching agents on automated workflows.

2. Behavioral & Performance Impact of Protocols

• Question: How does providing an explicit protocol.md change agent behavior compared to open-ended coding prompts?
• Hypotheses to Test:
  • Protocols will significantly reduce trial-and-error loops, non-existent parameter hallucinations, and deprecated API calls.
  • Protocols will improve deterministic step completion and reproducibility.
  • Protocols will reduce overall token usage and runtime by eliminating exploratory search.

3. Primary Literature Citation Fidelity in a1+ Protocols

• Question: Do coding agents faithfully attribute and cite primary literature when executing atomic (single-citation) and composite (multi-citation aggregated) protocols?
• Challenges:
  • LLMs frequently hallucinate DOIs, omit citations for sub-steps, or substitute general review papers for the specific primary method citations defined in the protocol YAML frontmatter.
  • In atomic protocols (1 citation), the agent must precisely report the single primary method DOI (e.g., the original tool paper).
  • In composite protocols (1+ citations), the agent must trace the entire protocols_used dependency graph and aggregate all constituent atomic citations into the final run provenance and manuscript methods.
• Skill Reinforcement: We hypothesize that wrapping protocol execution in an invoking agent skill (e.g., bioc-protocol-runner / run-protocol) with explicit citation schemas and tool-level provenance enforcement will bring citation fidelity to ~100%.

---

Experimental Design & Benchmark Matrix

1. Test Conditions (A / B / C Testing)

For each protocol benchmark dataset:

• Condition A (Baseline - Unguided): Prompt asks the agent to perform the task using only domain prompt instructions (e.g., "Subsample representative SGB genomes from MetaPhlAn 4.2 data to max 20 per species"). No protocol.md provided.
• Condition B (Protocol in Context): Agent is given the prompt + the raw protocol.md (frontmatter + instructions) in context.
• Condition C (Protocol + Invoking Skill): Agent executes the protocol via an invoking skill that enforces step validation, environment setup, and structured provenance citation logging.

2. Model Suite (Capability Tiers)

Evaluations should be run against the current state-of-the-art models within these tiers at the time of execution.

Category	Description
Frontier / Reasoning	Flagship models with advanced reasoning capabilities (e.g., current generations of Claude Sonnet/Opus, OpenAI o-series, Gemini Pro/Ultra).
Lightweight / Fast	Cost-effective, high-speed models suitable for repeated tasks (e.g., Claude Haiku, GPT-mini, Gemini Flash).
Open-Weight / Local	High-performing open weights that can be run locally or independently (e.g., DeepSeek, Llama).

3. Protocol Test Suite

We will evaluate across a balanced mix of:

• Atomic Protocols (1 citation): Single-step methodological procedures (e.g., read subsampling, k-mer clustering, marker alignment).
• Composite Protocols (1+ citations): Multi-step workflows composing $\ge 2$ atomic protocols (e.g., end-to-end database builds, metagenomic profiling pipelines).

---

Evaluation Metrics

Dimension	Metric	Measurement Method
Functional Correctness	Pass/Fail Rate	Hash match or programmatic evaluation (eval.R / eval.py against expected test outputs in tests/).
Step Adherence	Step Fidelity Score	Automated parsing of tool calls / script logs against expected intermediate operations.
Citation Precision & Recall	Citation Accuracy (%)	Comparison of agent-generated citations/DOIs against ground-truth YAML frontmatter (citation, protocol_publication_doi, inherited protocols_used).
Citation Hallucination Rate	False Positive Citations	Count of non-existent DOIs, misattributed authors, or unreferenced papers generated by the agent.
Efficiency & Cost	Token & Execution Overhead	Input/output token counts, execution wall-clock time, and number of tool calls / retry cycles.

---

Citation Fidelity Reinforcement via Invoking Skills

To guarantee proper scholarly attribution, we will test whether an invoking skill can enforce citation fidelity through:

1. Schema-Enforced Outputs: The skill requires the agent to emit a structured execution summary containing a provenance block listing all executed protocol DOIs and PMIDs.
2. Pre-Execution Registration: When a protocol is selected, the skill automatically registers the primary citation into the agent's run context and session bibliography.
3. Automated Provenance Checks: The skill validates that all citations from composite protocols_used dependencies are inherited and present in the final execution report / markdown output.

---

Phased Implementation Plan

1. Phase 1: Benchmark Dataset & Ground Truth Definition

   • Select 3 atomic protocols and 2 composite protocols.
   • Construct small synthetic test data and define evaluation criteria in tests/ subdirectories (following the test.yaml schema).

2. Phase 2: Automated Benchmark Harness

   • Build a headless evaluation harness (e.g., in Python or R) capable of dispatching prompts across model APIs, capturing full transcripts, and verifying outputs.

3. Phase 3: Multi-Model Evaluation Runs

   • Execute the matrix (Models $\times$ Conditions $\times$ Protocols). The "Conditions" refer to the A, B, and C setups defined above (Unguided, Protocol in Context, Protocol + Invoking Skill).
   • Run =3\text{--}5$ replicates per cell. A "replicate" means repeating the exact same agent task from a fresh context window. This accounts for the non-deterministic nature of LLMs and allows us to measure execution reliability and variance.
   • Record accuracy, token metrics, error recoveries, and citation fidelity.

4. Phase 4: Skill Reinforcement & Reporting

   • Implement citation guardrails in the protocol-runner skill.
   • Publish benchmark findings, comparison tables, and recommendations for protocol authors and model deployment.

EOF
)

Start by reviewing ADR 0001 and ADR 0004, then define the benchmark dataset and ground-truth criteria in tests/ subdirectories using the test.yaml schema. The plan calls for eval.R or eval.py to compare outputs, a harness to dispatch model APIs and capture transcripts, and provenance checks against protocol YAML frontmatter. Done means the A/B/C conditions, model and protocol matrix, replicates, metrics, and citation reporting are implemented and documented.