Formal Unit Testing and Benchmarks for Protocols

Summary

We propose extending the protocol specification to support formal unit tests and benchmarks via a standard tests/ subdirectory alongside protocol.md. The goal is to allow automated verification that an AI agent or human runner is correctly executing a protocol by supplying synthetic or real test data alongside expected outputs.

---

Motivation

As bioc-protocol-runner and other AI agents execute protocols autonomously, we need a way to verify whether execution by a particular agent and LLM model is correct.

• Verifiable Execution: If a protocol describes how to filter 16S reads, we need a small sample input and the exact expected output to confirm the agent used the right parameters and tools.
• Agent/Model Benchmarking: Benchmarks allow us to run CI/CD on the agents and LLMs themselves (i.e., testing which agent/model combinations successfully execute a protocol and which do not).
• Trust Tiering: Protocols that provide executable benchmarks can automatically earn a higher "Trust Tier" in federated registries, as their correctness is machine-verifiable.

---

Proposed Specification: The tests/ Subdirectory

To keep the system robust but simple across a growing federation of repositories, we propose a separated companion-file approach.

Tests will be stored in a standardized tests/ subdirectory within the protocol's folder. This prevents protocol.md from becoming bloated with binary file URLs and YAML blocks, and follows familiar software engineering paradigms (like R or Python package structures).

Example Structure:

protocols/humann4-augmented-clustering/
├── protocol.md
└── tests/
    ├── basic_filtering/
    │   ├── test.yaml          # Defines inputs, expected outputs, and eval scripts
    │   ├── eval.R             # Custom evaluation script (optional)
    │   ├── input_reads.fastq  # Small local synthetic data
    │   └── expected.fasta     # Expected output
    └── advanced_clustering/
        └── test.yaml          # A test relying on external URLs for larger data

1. Test Scope and Data Size

To avoid imposing unnecessary burdens on contributors:

• Data Size Flexibility: We recommend a soft limit (e.g., 5MB) for local test data within the repository to avoid git bloat. However, this is ultimately left up to individual federated repository owners to enforce based on their storage capacities. For larger datasets, test.yaml can define external download URLs.
• Test Scope: Test creators have full flexibility. A test can verify a single intermediate step (e.g., the output of Step 1) or the final terminal output of the entire end-to-end protocol.

2. Verification Logic (Deterministic vs. Probabilistic)

Because many bioinformatics tools are non-deterministic or probabilistic (meaning exact MD5 hashing of outputs will fail), the test framework will support two evaluation modes:

• Exact Hash Matching: For deterministic steps, check MD5/SHA256 hashes of the outputs against the expected files.
• Custom Evaluation Scripts: Provide an eval.R or eval.py script that parses the agent's output and determines if it passes (e.g., checking if clustering accuracy is >95%, or if a required subset of contigs is present).

3. Draft test.yaml Schema

We will start with a draft schema containing inputs, expected_outputs, and evaluation blocks. We will refine and finalize this schema in practice by building 2-3 proof-of-concept tests before freezing the standard.

---

Tooling & CI Impact

To support this across the federation, we will need to:

1. Agent Instructions & Guardrails: Because creating these detailed test setups is complex, contributors will heavily rely on agents to generate them. We must develop clear agent skills (e.g., create-protocol-benchmark) with strict guardrails to scaffold the test.yaml, mock data, and custom evaluation scripts automatically.
2. Execution Skill: Provide a test-protocol agent skill that parses test.yaml, loads inputs (local or remote), runs the protocol in a sandbox, and verifies the outputs (via hash or custom script).
3. validate-protocol.R: Validate the schema of any tests/*/test.yaml files present in a protocol's directory.
4. generate-protocols-yaml.R: Expose a has_benchmarks: true/false flag in the registry so users and agents know which protocols are machine-verifiable.

---

Next Steps

1. Draft the initial test.yaml JSON/YAML schema.
2. Implement 2-3 proof-of-concept tests for existing protocols to refine the schema and evaluation logic in practice.
3. Design the agent prompts and skill guardrails for automated test creation.

---

Co-authored-by: Antigravity gemini@google.com