Automattic / Automattic/data-liberation-agent
Capture has no proxy support: behind an intercepting network every import dies at the first fetch
- Dominant language
- TypeScript
- Stars
- 31
- Forks
- 3
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 81
Description
## Problem
DLA honors no proxy settings anywhere in the capture path. On a network where direct connections to the source site are intercepted or reset, every capture fails at the first fetch, and there is no configuration that can route around it.
Hit for real on 2 Sep 2026 while importing `https://www.kdclegal.com/`: the local network intercepts direct TLS to Wix's edge (Google Cloud LB) with a self-signed certificate. Every client fails on the direct path — Node fetch gets `ECONNRESET`, Chromium gets `ERR_CERT_AUTHORITY_INVALID`, plain curl fails validation — while the same requests through the environment's HTTP proxy (CONNECT tunnel) see the site's real, valid certificate and succeed. Three consecutive `studio create --from` runs died in under 6 seconds inside `cli.bundle.mjs`'s initial fetch. Any user behind a corporate TLS-inspecting or filtering proxy is in the same position.
## What already works, and what is missing
- **Node-side fetches** (platform detection, sitemap discovery, media fetch) need no code change: Node 24 honors `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` when `NODE_USE_ENV_PROXY=1` is set. Worth documenting.
- **Browser-side capture** is the gap. Every `chromium.launch()` in DLA (browser-kit, svg-raster, section-extract, fidelity check) launches with no proxy configuration, and Playwright does not read proxy env vars on its own. There is no flag, no env var, nothing.
## Fix contributed to PR #119
An env-gated launch option, `DLA_BROWSER_PROXY`:
- unset → launches exactly as before (zero behavior change);
- set to a proxy URL → every DLA-launched Chromium gets `proxy: { server, bypass: 'localhost,127.0.0.1,[::1]' }`.
TLS stays end-to-end through the CONNECT tunnel, so certificate validation against the real origin is untouched; localhost bypass keeps local replica servers and fidelity comparisons off the proxy. The 2 Sep kdclegal capture (65 routes, 499 MB) completed through this path on the first try after three direct-path failures.
Found while working the Wix migration gap list (run report: kdclegal, 7 Sep 2026).
Contributor guide
Research direction
Start by reviewing PR #119 and the browser-launching areas named in the issue: browser-kit, svg-raster, section-extract, and fidelity check. Verify how each Chromium launch handles DLA_BROWSER_PROXY, localhost bypass, and the unset case; done means browser capture works through the proxy without changing direct launches, while the Node-side proxy behavior is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, typescript
- Domain
- web-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100