Automattic / Automattic/skillsmith

More flexible scenario definition and judge-verified behavior

Open
#55 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Goal

A project using Skillsmith can describe what an implementation should *do* in plain language and have the judge (an AI agent) verify that behavior on a live site, instead of relying on hand-written e2e tests that assert specific selectors. The outcome is threefold:

1. **Verification reflects real behavior, not selector matching.** Today's e2e tests fail even when the testing agent's code is correct, because it's hard to always match the selectors the tests expect — a false failure that reflects test brittleness rather than wrong behavior. Asking the judge to check the intended behavior directly is also more flexible.
2. **Defining a scenario gets simpler.** Projects interact with Skillsmith through fewer, more self-contained files instead of today's web of linked rubrics, prompts, and spec files.
3. **Verification is no longer prescribed by Skillsmith.** Instead of baking in a fixed structure (rubrics + acceptance + e2e specs), a project expresses in prose whatever it wants the judge to check — so different projects can verify different things.

## Context

- **Our instance:** in the bundled `testing-project` we want the judge to check acceptance criteria, compare against rubrics, and verify UX behavior on a live site — but that is one project's choice, and the mechanism should stay general so other projects can ask for different things.
- **Scope spans two parts:** (a) a change to how projects interact with Skillsmith — the tool's surface for defining a scenario and its verification; and (b) updating the bundled `testing-project` to adopt the new approach and prove it works end-to-end.

## Assumptions / directions to explore

*(Open — later phases may confirm or revise the exact file split and naming.)*

- Replace the current per-scenario setup (`scenario.yaml` + linked rubrics + linked prompts + `e2e.spec.mjs`) with two self-contained prose files:
- **`TESTING-AGENT.md`** — which skills to use, the human prompt, and links to shared prompt files (e.g. the testing-agent prompt).
- **`JUDGE.md`** — what the judge analyzes: the acceptance criteria for evaluating the code, comparison against the rubrics, and an "e2e" section describing in plain language what to do on a live site (go to localhost, activate the plugin, check this behavior) — what the e2e tests cover today, but in human language.
- Folding rubrics and prompts into these files would remove the need to link them separately.

Contributor guide

Open the contributing guide

Research direction

Start by reading the current scenario setup, including scenario.yaml, the linked rubrics and prompts, and e2e.spec.mjs in the bundled testing-project. Compare that structure with the proposed TESTING-AGENT.md and JUDGE.md files, then trace how scenarios are defined and verified. Done means the bundled project uses the new approach and demonstrates judge-verified behavior end-to-end.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
developer-experience, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.