Config option to disable is_blocked() post-crawl content veto

is_blocked() (antibot_detector.py) is applied unconditionally in arun()'s post-processing and can mark successfully-fetched content as failed. Your own code already special-cases it twice in async_webcrawler.py — the comments note it "would misread 0 bytes html as a block" for binary downloads, and that real pages containing anti-bot script markers "trigger false positives" after fallback fetches. We hit a third case: small legitimate pages (a 378-byte file:// document) fail the minimal_text structural tier. Downstream consumers with their own quality pipelines currently have no way to receive the fetched content plus the verdict instead of a hard failure. Would you accept a PR adding a CrawlerRunConfig flag (e.g. check_blocked: bool = True) — or alternatively surfacing the verdict on a successful CrawlResult field — so callers can opt into making that judgment themselves?

Start by tracing arun() post-processing in async_webcrawler.py and the is_blocked() implementation in antibot_detector.py, including the existing binary-download and fallback-fetch special cases. Then inspect CrawlerRunConfig and CrawlResult to determine the least disruptive opt-out or verdict exposure, and add regression coverage showing that fetched content remains available for caller-side quality checks.