Automattic / Automattic/studio

AI site builds should not require screenshot self-validation by default

Open
#3,308 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
517
Forks
95
Avg merge
1d 23h
Merged PRs (30d)
172

Description

## Problem

The Studio AI system prompt currently requires visual self-validation as part of the default site-build workflow:

> Use `take_screenshot` to capture the site's landing page on desktop and mobile and verify the design visually on both viewports... Fix any issues found.

This makes every one-shot site generation pay for an agent-driven screenshot/vision review loop, even when the user can already see the generated site in Studio and can direct follow-up changes themselves.

## Why this matters

Mandatory screenshot self-validation adds latency and introduces extra failure modes to the critical path of site creation:

* The agent spends extra turns taking screenshots and reasoning over pixels instead of finishing promptly.
* Screenshot failures can make an otherwise-successful site build report as failed.
* The user-facing product already has a human in the loop who can review the result and request changes.
* Screenshot capture is still useful as an optional tool when explicitly requested, or when a harness/product layer wants evidence after completion.

## Evidence from BFB benchmark work

While testing the BFB Studio branch, the forced screenshot loop surfaced as a blocker unrelated to BFB conversion:

* The agent attempted `take_screenshot` during the default self-QA phase.
* One malformed `take_screenshot` call took \~1.6s before failing schema validation.
* One successful screenshot call took \~6.9s.
* The overall eval then timed out/aborted, so the screenshot failure poisoned the whole benchmark even though site generation had already made substantial progress.

Separate harness screenshots, taken after the agent run without involving the model loop, consistently took about 4-6.6s total for desktop + mobile in the same benchmark runs. That cost is measurable and can be paid by the product/harness when needed, without adding model turns or making screenshot success part of the agent's completion criteria.

## Proposed direction

Remove mandatory screenshot self-validation from the default AI site-build prompt.

Suggested replacement behavior:

* The agent should finish promptly after creating/updating the site and running cheap/static checks that are directly relevant.
* The user should review the visible result and direct refinements.
* `take_screenshot` should remain available for explicit user requests or non-interactive harnesses, but should not be required for every generated site.

This is separate from the narrower schema bug where `take_screenshot` should accept local-site `nameOrPath`/`path`: Automattic/studio#3306.

## AI assistance

* **AI assistance:** Yes
* **Tool(s):** OpenCode (GPT-5.5)
* **Used for:** Drafting this issue from benchmark evidence and observed Studio prompt/tool behavior.

Contributor guide

Open the contributing guide

Research direction

The issue names no file or test; start by locating the Studio AI system prompt that requires take_screenshot during site builds. Check the default workflow and keep screenshot use available for explicit requests or harnesses, while making completion depend on site creation or updates and relevant cheap/static checks rather than screenshot validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, desktop
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.