humanmade / humanmade/hm-rest-ability

Add a model-in-the-loop eval harness

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PHP
Stars
2
Forks
0
Avg merge
1d 12h
Merged PRs (30d)
16

Description

Protocol tests prove the tool works. They do not prove a model can work out how to use it. That is what the tool description and schema are for, and they regress quietly.

Add `tests/evals/`:

- `run.mjs` - the agent loop
- `scenarios/*.json` - a task prompt, setup steps, and post-conditions
- `grade.mjs` - checks post-conditions with direct REST calls

Rules:

- Build tool definitions from the live `tools/list`, so the eval tests what ships.
- Grade on WordPress state, not on what the model says. No LLM judge.
- Cap each run at 6 turns and 1024 output tokens. Truncate each tool result to about 4KB.
- Cache the system prompt and the tool block.

Default model is `claude-haiku-4-5`. Put the provider behind a small adapter so another model can be swapped in.

Write results to `test-results/evals.json`: pass or fail, turns, tokens and cost.

This issue covers local runs only. CI wiring is #18.

Depends on #15 and #16.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing protocol tests and the requirements in #15 and #16. Build the local harness under tests/evals/: run.mjs should use live tools/list, scenarios/*.json should define prompts and post-conditions, and grade.mjs should verify WordPress state through REST calls. Done means test-results/evals.json records pass or fail, turns, tokens, and cost for capped local runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, php, wordpress
Domain
ai, backend-api-design, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.