Automattic / Automattic/wp-codebox
Fail visual comparison when a captured document is an application error
- Dominant language
- TypeScript
- Stars
- 16
- Forks
- 4
- Avg merge
- 59m
- Merged PRs (30d)
- 131
Description
`wordpress.visual-compare` accepts a navigation as successful when the server responds with HTTP 200 but renders an application/runtime fatal page. The command then screenshots the error document and emits pixel-comparison evidence as though it represented the requested site.
## Reproduction
1. Serve an HTML response with status 200 whose body is a PHP fatal/error document.
2. Run `wordpress.visual-compare` against that URL.
3. Observe that navigation and screenshot capture complete without a page error.
## Expected
Before DOM snapshot or screenshot creation, visual capture should fail closed when the main-document response is unsuccessful or the rendered document is a bounded, recognized runtime-error page. Failure evidence should include the role/URL, HTTP status, and bounded reason without persisting a screenshot as valid comparison evidence. Normal page copy containing words such as “error” must remain valid.
## AI assistance
GPT-5.6 Sol via OpenCode reproduced the blind spot, traced it to the visual-compare capture boundary, and drafted this issue. Chris Huber remains responsible for the report and resulting changes.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the wordpress.visual-compare capture boundary and reproduce the documented HTTP 200 PHP fatal/error response. Trace navigation through DOM snapshot and screenshot creation; done means recognized runtime-error documents fail closed with role, URL, status, and a bounded reason, while normal page text containing “error” remains valid and no invalid screenshot is persisted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100