waldronlab / waldronlab/agent-protocol-standard

Check that the runner's field list matches the standard, and its own steps

Open
#16 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3
Forks
1
Avg merge
14h 8m
Merged PRs (30d)
23

Description

skills/protocol-runner/SKILL.md step 1.2 enumerates the PROTOCOLS.yaml fields the runner relies on. Nothing guarantees that list still matches the frontmatter schema in PROTOCOL_STANDARD.md.

This is not hypothetical — the Copilot review of #14 found the list wrongly included trust_tier, which is a property of a repository's entry in registry.yaml, not of a protocol. An agent following the steps literally had nothing to rank candidates with. That bug was present from the skill's first release and survived because no check compares the two documents.

A small script, runnable from tests/run-tests.R, could parse both and fail when the skill names a field the standard does not define (or misses a required one).

Recorded in ADR 0007 §5 as the check worth having, in place of reproducing Bioconductor's more general validate_skills.py for a single skill.


Widened 2026-09-13: a two-way comparison passes today while the skill is wrong.

Checking the skill against the standard catches a field that doesn't exist. It misses a field the skill omits from its own list and then uses anyway, which is the live bug:

  • step 1.2 lists the fields "this skill relies on", omitting category and tags
  • step 2.2 matches the user's task on name, description, category, and tags

Both are defined in the standard, so comparing the two documents passes. An agent that fetched only the listed fields has nothing to match on. Same shape at the Method Provenance block, which prints Following: [Author] ... while authors isn't in the list either.

So the script should fail when:

  1. the skill names a field PROTOCOL_STANDARD.md doesn't define
  2. the skill omits a field the standard requires
  3. the skill's own later steps use a field step 1.2 doesn't list

Only the third catches what's broken now.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read skills/protocol-runner/SKILL.md, PROTOCOL_STANDARD.md, tests/run-tests.R, and ADR 0007 §5 first. Add a check that fails for undefined or omitted required fields and for fields used by later skill steps but absent from step 1.2, then run the test entry point to verify the current omissions are detected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
documentation, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.