Automattic / Automattic/wp-codebox

Fail visual comparison when a captured document is an application error

Open
#2,256 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

`wordpress.visual-compare` accepts a navigation as successful when the server responds with HTTP 200 but renders an application/runtime fatal page. The command then screenshots the error document and emits pixel-comparison evidence as though it represented the requested site.

## Reproduction

1. Serve an HTML response with status 200 whose body is a PHP fatal/error document.
2. Run `wordpress.visual-compare` against that URL.
3. Observe that navigation and screenshot capture complete without a page error.

## Expected

Before DOM snapshot or screenshot creation, visual capture should fail closed when the main-document response is unsuccessful or the rendered document is a bounded, recognized runtime-error page. Failure evidence should include the role/URL, HTTP status, and bounded reason without persisting a screenshot as valid comparison evidence. Normal page copy containing words such as “error” must remain valid.

## AI assistance

GPT-5.6 Sol via OpenCode reproduced the blind spot, traced it to the visual-compare capture boundary, and drafted this issue. Chris Huber remains responsible for the report and resulting changes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the wordpress.visual-compare capture boundary and reproduce the documented HTTP 200 PHP fatal/error response. Trace navigation through DOM snapshot and screenshot creation; done means recognized runtime-error documents fail closed with role, URL, status, and a bounded reason, while normal page text containing “error” remains valid and no invalid screenshot is persisted.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.