web-infra-dev / web-infra-dev/midscene

[Bug]: HTML report shows Playground demo data (SauceDemo) instead of real UI context after aiAssert

Open
#3,041 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
14.9k
Forks
1.2k
Avg merge
1d 14h
Merged PRs (30d)
96

Description

Version
  • @midscene/core: 1.12.0
  • @midscene/web: 1.12.0
  • Playwright: 1.62.1
  • Node.js: 24.12.0
  • macOS (darwin)
Describe the bug

When running Midscene.js with Playwright (via @midscene/web/playwright PlaywrightAiFixture), AI actions (aiAssert) execute correctly against the real page — the AI model's analysis text is correct (e.g. "a gray toast ... displayed in the center of the page" → StatementIsTruthy: true). The screenshot is also captured successfully at runtime (screenshotBase64 begin/end in web-page.log).

However, the generated HTML report does not contain the real UI context (screenshot + DOM). Instead, the report shows Playground demo data — a SauceDemo page (Swag Labs, Username, Login, etc.) as the UI context preview.

Steps to reproduce
  1. Set up Playwright + Midscene (PlaywrightAiFixture).
  2. Write a test that calls aiAssert(...) on any page (e.g. mobile dispatch page).
  3. Run the test, let aiAssert execute successfully.
  4. Open the generated HTML report under midscene_run/report/.
Expected behavior

The report should show the real screenshot + DOM context captured at runtime.

Actual behavior

The report shows SauceDemo demo data instead of the real page context.

Root cause analysis (from source)
  • packages/visualizer/src/component/context-preview/index.tsx renders No UI context + demo data when no real UI context is provided to the report.
  • The demo data is packages/visualizer/src/component/playground/playground-demo-ui-context.json (SauceDemo Swag Labs).
  • AI runtime DOES capture the screenshot (confirmed in midscene_run/log/web-page.log: screenshotBase64 begin/end), and the model's analysis text references the real screenshot correctly (midscene_run/log/ai-call.log).
  • But the report data pipeline seems to lose the UI context (screenshot) before it reaches the HTML reporter, so the report falls back to demo data.
Additional notes
  • midscene_run/log/cache.log shows no cache file found every run — cache is not the cause.
  • The report file is freshly generated (current timestamp), not stale.
  • This affects @midscene/web/playwright playwright reporter separate mode with default single-html output.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/visualizer/src/component/context-preview/index.tsx and the PlaywrightAiFixture report path, then reproduce the issue using the steps and compare web-page.log, ai-call.log, and the generated report. Trace where the captured screenshot and DOM are lost before HTML generation. Done means the report shows the real page context instead of the SauceDemo data from playground-demo-ui-context.json.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.