Automattic / Automattic/data-liberation-agent

Capture has no proxy support: behind an intercepting network every import dies at the first fetch

Open
#184 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
31
Forks
3
Avg merge
10h 14m
Merged PRs (30d)
81

Description

## Problem

DLA honors no proxy settings anywhere in the capture path. On a network where direct connections to the source site are intercepted or reset, every capture fails at the first fetch, and there is no configuration that can route around it.

Hit for real on 2 Sep 2026 while importing `https://www.kdclegal.com/`: the local network intercepts direct TLS to Wix's edge (Google Cloud LB) with a self-signed certificate. Every client fails on the direct path — Node fetch gets `ECONNRESET`, Chromium gets `ERR_CERT_AUTHORITY_INVALID`, plain curl fails validation — while the same requests through the environment's HTTP proxy (CONNECT tunnel) see the site's real, valid certificate and succeed. Three consecutive `studio create --from` runs died in under 6 seconds inside `cli.bundle.mjs`'s initial fetch. Any user behind a corporate TLS-inspecting or filtering proxy is in the same position.

## What already works, and what is missing

- **Node-side fetches** (platform detection, sitemap discovery, media fetch) need no code change: Node 24 honors `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` when `NODE_USE_ENV_PROXY=1` is set. Worth documenting.
- **Browser-side capture** is the gap. Every `chromium.launch()` in DLA (browser-kit, svg-raster, section-extract, fidelity check) launches with no proxy configuration, and Playwright does not read proxy env vars on its own. There is no flag, no env var, nothing.

## Fix contributed to PR #119

An env-gated launch option, `DLA_BROWSER_PROXY`:

- unset → launches exactly as before (zero behavior change);
- set to a proxy URL → every DLA-launched Chromium gets `proxy: { server, bypass: 'localhost,127.0.0.1,[::1]' }`.

TLS stays end-to-end through the CONNECT tunnel, so certificate validation against the real origin is untouched; localhost bypass keeps local replica servers and fidelity comparisons off the proxy. The 2 Sep kdclegal capture (65 routes, 499 MB) completed through this path on the first try after three direct-path failures.

Found while working the Wix migration gap list (run report: kdclegal, 7 Sep 2026).

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #119 and the browser-launching areas named in the issue: browser-kit, svg-raster, section-extract, and fidelity check. Verify how each Chromium launch handles DLA_BROWSER_PROXY, localhost bypass, and the unset case; done means browser capture works through the proxy without changing direct launches, while the Node-side proxy behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.