agentic evals

Phase 1

• input a prompt ("find me the leaderboard page and tell me if it has a bar chart on it showing actual data")
  • including hints about how to find it (url, etc)
• generate the stagehand steps to get to the right place
• run the steps
• structured output PASS/FAIL

Phase 2

• the "steps" are dynamic: as the model is trying each step, its adding or removing to the checklist, and maybe taking notes on what worked.
• retying different strategies of finding the right place
• since these actions are very quick, it can retry lots of times.

Phase 3

• fixtures: storing sequences of steps that can be re-used later by other runs or other agents
• Framework for a human to click through the UI and record the elements names, to create "fixtures"

Full Agentic

• Dynamic evolving MCP server for each repo
• Starts off with the simple 7 stagehand tools
• But as fixtures are discovered or added by a human, they are added as MCP tools
• Agent just loops and discovers for itself. No intermediate "steps" representation

No files, tests, or entry points are named. Start by reviewing how the repository currently uses Stagehand, then define the Phase 1 boundary: prompt and hints in, generated steps executed, and structured PASS/FAIL out. Completion criteria for the later dynamic, fixture, and MCP phases would need to be agreed before implementation.