adobe / adobe/skills

fix(scrape-webpage): harden analyze-webpage.js against bot detection and HTTP/2 errors

Open
#82 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
182
Forks
73
Avg merge
1d 5h
Merged PRs (30d)
30

Description

## Summary

`analyze-webpage.js` uses a hardcoded Chromium executable path and a fragile browser
configuration that can be blocked by sites that detect headless browsers (e.g. HTTP/2
rejection, bot fingerprinting).

## Changes needed

- **Replace hardcoded `executablePath`** (`/ms-playwright/chromium-1208/chrome-linux/chrome`)
with dynamic Chrome-first auto-detection: try `channel: 'chrome'` first (real TLS
fingerprint), fall back to bundled Chromium silently. The hardcoded path breaks if the
Chromium version changes and doesn't work outside Docker.
- **Add `--disable-http2`** to browser args to prevent `ERR_HTTP2_PROTOCOL_ERROR` from
servers that reject HTTP/2 connections from headless browsers.
- **Switch navigation to `domcontentloaded`** (60s timeout + 5s settle) instead of
`networkidle`. Many sites never reach networkidle and the script times out unnecessarily.
- **Add explicit `timeout: 60000`** to the screenshot call to prevent failures on large pages.
- **Align browser context config** with `run-bulk-import.js`: Chrome 131 UA, realistic
`sec-ch-ua` headers, locale, timezone, `ignoreHTTPSErrors`.

Contributor guide

Open the contributing guide

Research direction

Start in analyze-webpage.js and compare its browser launch and context configuration with run-bulk-import.js. Check the navigation and screenshot options first, then verify Chrome-first detection falls back silently to bundled Chromium and that the requested headers, locale, timezone, HTTP/2 setting, timeouts, and DOM-content-loaded flow are all applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
web-dev
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.