Feature request: opt-in block_internal_urls egress filter for the library layer

Thanks for following up from the email thread, @ntohidi.

Quick correction on my side: the issue body initially only contained a literal file path — I mistakenly relied on @path expansion with gh api, which doesn't expand files (that's a curl / --body-file flag). Pasting the real content here.

Alignment

We're aligned on the classification. The library is a user agent invoked by a trusted caller, so destination filtering should stay opt-in, and the SSRF trust boundary remains at the Docker API server where egress_broker.py already enforces it. The agentic / LLM-chosen-URL case is the scenario that justifies exposing the same primitives to library callers.

Proposed design

• Flag: block_internal_urls: bool (default False, opt-in). Set per-crawl so callers who embed Crawl4AI in an agent can opt in without a global change.
• Chokepoint: a single host-validation call inserted right after the existing scheme allow-list check in AsyncCrawlerStrategy.crawl() (crawl4ai/async_crawler_strategy.py). It must cover both egress paths:
  • HTTP path: AsyncHTTPCrawlerStrategy._handle_http() (aiohttp)
  • Browser path: browser_manager.py → Playwright goto()
• Logic reuse: port the existing validate_url_destination + resolve_and_pin (DNS pinning) + per-hop redirect revalidation from deploy/docker/utils.py / egress_broker.py into a shared helper (e.g. crawl4ai/url_safety.py) so the library and the Docker server share one implementation — no duplicated trust logic.
• Blocked ranges: loopback (127.0.0.0/8, ::1), private (10/8, 172.16/12, 192.168/16, fc00::/7), link-local (169.254/16 incl. cloud metadata 169.254.169.254, fe80::/10), and 0.0.0.0/8. The resolved IP must be checked after DNS (pin) and after every redirect hop (revalidate) to prevent DNS-rebinding / redirect-to-internal bypasses.
• Behavior on block: raise a BlockedURL exception (or return a failed CrawlResult with a clear error) rather than fetching.

Open questions (happy to match maintainer preference)

1. Flag name — block_internal_urls vs deny_private_destinations vs egress_filter?
2. Where the chokepoint sits — shared base crawl() vs per-strategy hooks?
3. Browser redirect following — should the redirect revalidation also cover hops taken by Playwright goto, or only the initial URL?

I'd be happy to draft the PR implementing this once we settle the surface.

Read crawl4ai/async_crawler_strategy.py and trace the HTTP path through AsyncHTTPCrawlerStrategy._handle_http() and the browser path through browser_manager.py to Playwright goto(). Then compare validate_url_destination and resolve_and_pin in deploy/docker/utils.py and egress_broker.py; the work is done when one shared opt-in filter covers DNS resolution, redirects, and both egress paths without fetching blocked destinations.