Playwright protocol applies neither the IP filter nor the URL filters to the requests it makes
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
## What happens
`IPFilterRules` is only used by the okhttp protocol, where it runs as a network interceptor. The playwright protocol never constructs it, so `http.filter.ipaddress.include` and `http.filter.ipaddress.exclude` have no effect on a topology that fetches with playwright. The module already installs a route handler that sees every request the browser makes, but that handler only aborts resource types listed in `playwright.skip.resource.types` and otherwise resumes. This covers the navigation itself, its redirect hops, and every subresource, XHR, iframe and script request the rendered page issues.
## Where
`external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:243-256`.
```java
page.route( lambdaUrl -> true, route -> { // abort if we know the main page is a redirection if (status.get() != -1) { ... } else if (resourceTypesToSkip.contains(route.request().resourceType())) { route.abort(); } else { route.resume(); } });
```
The IP filter itself lives in `core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java` and is wired up only in `core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java`.
## Why it matters
This is documented: `crawler-default.yaml:161-162` and `docs/src/main/asciidoc/configuration.adoc:259-260` both say the IP filter is okhttp only, and both keys ship commented out, so nobody was told something untrue. The gap is still worth closing. A crawler that renders pages in a browser makes far more requests than one that fetches bytes, and every one of them leaves the worker with no address check, including requests the fetched page decides to make. An operator who moves a topology from okhttp to playwright loses the control silently, because the keys stay in the configuration and stop doing anything.
## Reproduction
No automated test. `configure()` launches or connects to a real Chromium, so exercising the route handler needs a browser plus an HTTP server bound to a filtered address.
Manual steps:
1. Configure a topology with the playwright protocol and set `http.filter.ipaddress.exclude: "localhost,sitelocal"`.
2. Run a small HTTP server on the loopback interface and fetch its URL through the protocol. It is fetched, whereas the okhttp protocol fails the fetch with an `IOException`.
3. Serve a page whose body requests a subresource from a loopback URL. The request is issued by the browser and the route handler resumes it.
## Suggested fix
Build an `IPFilterRules` instance in `HttpProtocol.configure()` from the same configuration keys, and apply it inside the existing `page.route` handler: resolve the request host and call `route.abort()` when the address is rejected. Apply `URLFilters` in the same place so redirect hops and subresource requests are checked against the crawl scope. Update `docs/src/main/asciidoc/configuration.adoc` so the "OkHttp only" note reflects the new state. Resolution in the handler adds a lookup per request, so cache results per host. A separate DNS lookup is also not the address the browser ends up connecting to, which is worth stating in the documentation.
Contributor guide
Research direction
Start in external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:243-256 and inspect configure() and the existing page.route handler. Compare core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java and the okhttp HttpProtocol.java, then manually reproduce filtered navigation and subresource requests with a real Chromium and HTTP server. Done means IP and URL filters apply to browser requests and the OkHttp-only documentation is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, playwright
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100