web-infra-dev / web-infra-dev/midscene

[Bug]: Report screenshots are completely black in headless Linux environment

Open
#2,970 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
14.9k
Forks
1.2k
Avg merge
1d 14h
Merged PRs (30d)
96

Description

Version
Environment
@midscene/core: 1.10.3
@midscene/web: 1.10.3
Playwright: 1.48.0
Node.js: 22.23.1
OS: Ubuntu 22.04 LTS (headless, no GPU)
Browser: Chromium (headless mode)
MIDSCENE_OUTPUT_FORMAT: single-html
Details

Describe the bug

When running Midscene.js with Playwright in a headless Linux environment (no GPU), the AI agent correctly identifies and interacts with page elements, but the screenshots saved in the HTML report are completely black.

Environment

  • @midscene/core: 1.10.3
  • @midscene/web: 1.10.3
  • Playwright: 1.48.0
  • Node.js: 22.23.1
  • OS: Ubuntu 22.04 LTS (headless, no GPU)
  • Browser: Chromium (headless mode)
  • MIDSCENE_OUTPUT_FORMAT: single-html

Steps to Reproduce

  1. Run a Midscene script on a Linux server with headless Chromium
  2. The target page has CSS animations and gradient backgrounds (a login page with animated background)
  3. Script uses agent.aiTap(), agent.aiAssert() etc.
  4. All AI actions succeed — the agent correctly locates buttons, input fields, and verifies page content
  5. Open the generated HTML report

Expected Behavior

The HTML report should show screenshots of the page at each step, matching what the AI "saw" during execution.

Actual Behavior

All screenshots in the report are completely black. The HTML file is 3-4MB (screenshots ARE embedded as base64 in single-html mode), but the image content is black.

Key Observation

The AI agent can correctly identify elements on the page (e.g., "账号输入框" at coordinates [1330, 346], "登录按钮", etc.). This means the screenshots taken for AI processing are rendered correctly. However, the screenshots saved to the report are black.

This suggests that the screenshots used for AI inference and the screenshots saved to the report are captured at different times or through different mechanisms, and the report screenshots fail to render properly.

What I've Tried

  1. --disable-gpu flag → screenshots still black
  2. --use-angle=swiftshader flag → screenshots completely broken (file not found)
  3. MIDSCENE_OUTPUT_FORMAT=single-html → screenshots embedded but still black
  4. MIDSCENE_OUTPUT_FORMAT=html-and-external-assets → external PNG files not generated at all
  5. Adding waitForTimeout(3000) before actions → no effect on report screenshots

Script Example

page.setViewportSize({ width: 1600, height: 900 });
await page.goto('xxxxx');
await page.waitForTimeout(3000);
await page.getByText('账号密码登录').click();
await page.waitForTimeout(1000);
await agent.aiTap('账号输入框');
await page.keyboard.type('xxx');
await agent.aiTap('密码输入框');
await page.keyboard.type('password');
await agent.aiTap('登录按钮');
await page.waitForTimeout(3000);

### Reproduce link

N/A

### Reproduce Steps

1. Set up a Linux server (Ubuntu 22.04) with Node.js 22+ and no GPU
2. Install Chromium via Playwright: npx playwright install chromium
3. Install Midscene: npm install @midscene/core @midscene/web
4. Create a script that uses PlaywrightAgent to interact with a page that has CSS animations/gradients:

const { chromium } = require('playwright');
const { PlaywrightAgent } = require('@midscene/web/playwright');

process.env.MIDSCENE_OUTPUT_FORMAT = 'single-html';
process.env.MIDSCENE_MODEL_NAME = 'your-model';
process.env.MIDSCENE_MODEL_API_KEY = 'your-key';
process.env.MIDSCENE_MODEL_FAMILY = 'your-family';

(async () => {
  const browser = await chromium.launch({ headless: true, args: ['--no-sandbox'] });
  const page = await browser.newPage({ viewport: { width: 1600, height: 900 } });
  const agent = new PlaywrightAgent(page);
  
  await page.goto('xxxx');
  await page.waitForTimeout(3000);
  await agent.aiTap('账号输入框');
  await page.keyboard.type('test');
  
  // Check the generated HTML report - screenshots will be black
  await browser.close();
})();

5. Open the generated HTML report in a browser
6. Observe that all screenshots in the report are completely black, even though AI correctly identified elements

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the PlaywrightAgent entry point and the single-html report path, comparing the screenshots used for AI inference with those embedded in the report. Reproduce on headless Ubuntu with Chromium and confirm that report images are rendered rather than black while AI actions still succeed.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, playwright, typescript
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.