apache / apache/stormcrawler

Playwright browser context stops validating certificates as soon as a proxy is configured

Open
#2,094 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

## What happens
`HttpProtocol.configure()` builds the browser context options and, when `http.proxy` is set, it calls `setIgnoreHTTPSErrors(true)` right after `setProxy()`. The two calls are tied together: there is no configuration key that controls the flag, and no way to keep certificate validation on while fetching through a proxy. The flag is a `NewContextOptions` setting, so it applies to the whole context: every page, every navigation and every subresource, not just the leg that terminates at the proxy. Without a proxy the browser validates normally, so the behaviour flips on an unrelated setting.

## Where
`external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:162-170`, config keys `http.proxy`, `http.proxy.username`, `http.proxy.password`.

```java
final Proxy globalProxy = getProxy(proxyServer, proxyUser, proxyPwd);
if (globalProxy != null) { b_c_options.setProxy(globalProxy); b_c_options.setIgnoreHTTPSErrors(true);
}
```

## Why it matters
An operator who configures a proxy gets a browser that renders any HTTPS page whose certificate does not verify, and the browser executes the JavaScript on that page. Nothing in `playwright-conf.yaml` or the module README mentions the behaviour, so the operator has no reason to expect it and no key to turn it off. Core has a comparable setting for okhttp (`http.trust.everything`, documented in configuration.adoc), which is exactly what is missing here: a named, documented switch rather than a side effect. The exposure needs someone able to intercept or answer for the target host, so it only bites deployments where the network path between the proxy and the origin is not trusted.

## Reproduction
No automated test. `configure()` launches or connects to a real Chromium instance, and the flag only shows up in the browser's TLS handshake, so reproducing it needs a browser and a TLS endpoint. The module's own `ProtocolTest` already requires a running Chrome for the same reason.

Manual steps:

1. Serve any page over HTTPS with a certificate the container does not trust (for example a self-signed one).
2. Configure the playwright protocol without a proxy and fetch the URL. The fetch fails on the certificate.
3. Set `http.proxy` to a proxy that forwards to the same host and fetch again. The page is fetched and rendered.

## Suggested fix
Introduce a dedicated key, for example `playwright.ignore.https.errors`, read it in `configure()` and pass it to `setIgnoreHTTPSErrors()` independently of whether a proxy is set. Default it to false and log a warning when it is enabled. Document it in `playwright-conf.yaml` next to the other playwright keys. Note the compatibility impact: deployments running behind a TLS-intercepting egress proxy currently depend on the implicit behaviour and will need either the new key or the proxy CA installed in the browser image.

Contributor guide

Open the contributing guide

Research direction

Start in external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:162-170 and inspect how the existing http.proxy keys are read. Review playwright-conf.yaml, the module README, configuration.adoc, and ProtocolTest before reproducing with a running Chrome and an HTTPS endpoint. Done means certificate validation is independently controlled by a documented key, defaults to false, and warns when enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.