humanmade / humanmade/hm-rest-ability
Add a model-in-the-loop eval harness
- Dominant language
- PHP
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 16
Description
Protocol tests prove the tool works. They do not prove a model can work out how to use it. That is what the tool description and schema are for, and they regress quietly.
Add `tests/evals/`:
- `run.mjs` - the agent loop
- `scenarios/*.json` - a task prompt, setup steps, and post-conditions
- `grade.mjs` - checks post-conditions with direct REST calls
Rules:
- Build tool definitions from the live `tools/list`, so the eval tests what ships.
- Grade on WordPress state, not on what the model says. No LLM judge.
- Cap each run at 6 turns and 1024 output tokens. Truncate each tool result to about 4KB.
- Cache the system prompt and the tool block.
Default model is `claude-haiku-4-5`. Put the provider behind a small adapter so another model can be swapped in.
Write results to `test-results/evals.json`: pass or fail, turns, tokens and cost.
This issue covers local runs only. CI wiring is #18.
Depends on #15 and #16.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing protocol tests and the requirements in #15 and #16. Build the local harness under tests/evals/: run.mjs should use live tools/list, scenarios/*.json should define prompts and post-conditions, and grade.mjs should verify WordPress state through REST calls. Done means test-results/evals.json records pass or fail, turns, tokens, and cost for capped local runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, php, wordpress
- Domain
- ai, backend-api-design, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100