web-infra-dev / web-infra-dev/midscene
[Bug]: Report screenshots are completely black in headless Linux environment
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 14.9k
- Forks
- 1.2k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 96
Description
Version
Environment
@midscene/core: 1.10.3
@midscene/web: 1.10.3
Playwright: 1.48.0
Node.js: 22.23.1
OS: Ubuntu 22.04 LTS (headless, no GPU)
Browser: Chromium (headless mode)
MIDSCENE_OUTPUT_FORMAT: single-html
Details
Describe the bug
When running Midscene.js with Playwright in a headless Linux environment (no GPU), the AI agent correctly identifies and interacts with page elements, but the screenshots saved in the HTML report are completely black.
Environment
@midscene/core: 1.10.3@midscene/web: 1.10.3- Playwright: 1.48.0
- Node.js: 22.23.1
- OS: Ubuntu 22.04 LTS (headless, no GPU)
- Browser: Chromium (headless mode)
MIDSCENE_OUTPUT_FORMAT:single-html
Steps to Reproduce
- Run a Midscene script on a Linux server with headless Chromium
- The target page has CSS animations and gradient backgrounds (a login page with animated background)
- Script uses
agent.aiTap(),agent.aiAssert()etc. - All AI actions succeed — the agent correctly locates buttons, input fields, and verifies page content
- Open the generated HTML report
Expected Behavior
The HTML report should show screenshots of the page at each step, matching what the AI "saw" during execution.
Actual Behavior
All screenshots in the report are completely black. The HTML file is 3-4MB (screenshots ARE embedded as base64 in single-html mode), but the image content is black.
Key Observation
The AI agent can correctly identify elements on the page (e.g., "账号输入框" at coordinates [1330, 346], "登录按钮", etc.). This means the screenshots taken for AI processing are rendered correctly. However, the screenshots saved to the report are black.
This suggests that the screenshots used for AI inference and the screenshots saved to the report are captured at different times or through different mechanisms, and the report screenshots fail to render properly.
What I've Tried
--disable-gpuflag → screenshots still black--use-angle=swiftshaderflag → screenshots completely broken (file not found)MIDSCENE_OUTPUT_FORMAT=single-html→ screenshots embedded but still blackMIDSCENE_OUTPUT_FORMAT=html-and-external-assets→ external PNG files not generated at all- Adding
waitForTimeout(3000)before actions → no effect on report screenshots
Script Example
page.setViewportSize({ width: 1600, height: 900 });
await page.goto('xxxxx');
await page.waitForTimeout(3000);
await page.getByText('账号密码登录').click();
await page.waitForTimeout(1000);
await agent.aiTap('账号输入框');
await page.keyboard.type('xxx');
await agent.aiTap('密码输入框');
await page.keyboard.type('password');
await agent.aiTap('登录按钮');
await page.waitForTimeout(3000);
### Reproduce link
N/A
### Reproduce Steps
1. Set up a Linux server (Ubuntu 22.04) with Node.js 22+ and no GPU
2. Install Chromium via Playwright: npx playwright install chromium
3. Install Midscene: npm install @midscene/core @midscene/web
4. Create a script that uses PlaywrightAgent to interact with a page that has CSS animations/gradients:
const { chromium } = require('playwright');
const { PlaywrightAgent } = require('@midscene/web/playwright');
process.env.MIDSCENE_OUTPUT_FORMAT = 'single-html';
process.env.MIDSCENE_MODEL_NAME = 'your-model';
process.env.MIDSCENE_MODEL_API_KEY = 'your-key';
process.env.MIDSCENE_MODEL_FAMILY = 'your-family';
(async () => {
const browser = await chromium.launch({ headless: true, args: ['--no-sandbox'] });
const page = await browser.newPage({ viewport: { width: 1600, height: 900 } });
const agent = new PlaywrightAgent(page);
await page.goto('xxxx');
await page.waitForTimeout(3000);
await agent.aiTap('账号输入框');
await page.keyboard.type('test');
// Check the generated HTML report - screenshots will be black
await browser.close();
})();
5. Open the generated HTML report in a browser
6. Observe that all screenshots in the report are completely black, even though AI correctly identified elements
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the PlaywrightAgent entry point and the single-html report path, comparing the screenshots used for AI inference with those embedded in the report. Reproduce on headless Ubuntu with Chromium and confirm that report images are rendered rather than black while AI actions still succeed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, playwright, typescript
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100