fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors
- Dominant language
- JavaScript
- Stars
- 182
- Forks
- 73
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 30
Description
## Summary
`analyze-webpage.js` uses a hardcoded Chromium executable path and a fragile browser
configuration that can be blocked by sites that detect headless browsers (e.g. HTTP/2
rejection, bot fingerprinting).
## Changes needed
- **Replace hardcoded `executablePath`** (`/ms-playwright/chromium-1208/chrome-linux/chrome`)
with dynamic Chrome-first auto-detection: try `channel: 'chrome'` first (real TLS
fingerprint), fall back to bundled Chromium silently. The hardcoded path breaks if the
Chromium version changes and doesn't work outside Docker.
- **Add `--disable-http2`** to browser args to prevent `ERR_HTTP2_PROTOCOL_ERROR` from
servers that reject HTTP/2 connections from headless browsers.
- **Switch navigation to `domcontentloaded`** (60s timeout + 5s settle) instead of
`networkidle`. Many sites never reach networkidle and the script times out unnecessarily.
- **Add explicit `timeout: 60000`** to the screenshot call to prevent failures on large pages.
- **Align browser context config** with `run-bulk-import.js`: Chrome 131 UA, realistic
`sec-ch-ua` headers, locale, timezone, `ignoreHTTPSErrors`.
Contributor guide
Research direction
Start in analyze-webpage.js and compare its browser launch and context configuration with run-bulk-import.js. Check the navigation and screenshot options first, then verify Chrome-first detection falls back silently to bundled Chromium and that the requested headers, locale, timezone, HTTP/2 setting, timeouts, and DOM-content-loaded flow are all applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100