unclecode / unclecode/crawl4ai

Feature Request: Auto-Follow Single-Frame Framesets

Open
#1,974 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

✨ Enhancement
Dominant language
Python
Stars
83.9k
Forks
8.7k
Avg merge
3d 7h
Merged PRs (30d)
11

Description

Summary

When crawling a URL that serves an HTML <frameset> page, crawl4ai returns the empty frameset shell instead of the actual content, then incorrectly reports the result as blocked by antibot protection. The URL is perfectly accessible — the content simply lives inside the frame, which Playwright has already loaded but crawl4ai does not read.

Reproduction

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
import asyncio

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://www.variosolar.de/")
        print(result.success)        # False
        print(result.error_message)  # "Blocked by anti-bot protection: ..."
        print(len(result.html))      # 612

asyncio.run(main())

Output:

[ERROR] Blocked by anti-bot protection: Structural: minimal_text on small page (612 bytes, 32 chars visible)
False
612

Opening https://www.variosolar.de/ in a browser shows a fully rendered, content-rich page with no antibot challenge whatsoever.

Root Cause

variosolar.de responds with this HTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" ...>
<html>
  <head><title>VarioSolar</title></head>
  <frameset rows="100%">
    <frame src="https://www.varioself.de/bauweise/varioenergy/" frameborder="0" noresize="noresize">
  </frameset>
</html>

This is a frameset redirect — a pattern from the early web where a site acts as a transparent wrapper pointing all traffic to another URL. A browser renders it invisibly: the user sees the frame content, the frameset shell is never visible.

crawl4ai calls page.content() on the top-level page after load, which returns the frameset HTML (612 bytes). The frame content — which Playwright has already fully loaded in a child frame — is never read. The antibot detector then sees a nearly empty page and fires a false positive.

Why Existing Config Options Don't Help

Option Why it fails
process_iframes=True Handles <iframe> elements only. <frameset>/<frame> are a distinct, older HTML construct not touched by this code path.
remove_overlay_elements=True Irrelevant — the problem is missing content, not overlays.
wait_until="networkidle" The frameset shell loads instantly. Waiting longer returns the same 612 bytes.
delay_before_return_html Same reason — time does not change what page.content() returns.

Workaround (manual): Inspect the raw HTML of the frameset page, extract the src attribute from the <frame> tag, and crawl that URL directly:

# Instead of the user-facing URL:
result = await crawler.arun("https://www.variosolar.de/")          # ✗ empty shell

# Crawl the actual framed URL directly:
result = await crawler.arun("https://www.varioself.de/bauweise/varioenergy/")  # ✓ works

This is a poor user experience: the site looks completely normal in a browser, the error message is actively misleading, and finding the real URL requires manually inspecting raw HTML.

Proposed Fix: follow_frames on CrawlerRunConfig

Behaviour

Add a follow_frames parameter to CrawlerRunConfig. When enabled, crawl4ai detects single full-viewport frameset pages after load and transparently returns the child frame's content and URL — exactly as HTTP 301/302 redirects are already followed automatically.

The key insight: Playwright has already loaded and rendered the frame content. It is accessible via page.frames at no extra cost. No second HTTP request is needed.

Detection Heuristic

Qualify a page as a frameset redirect only if all conditions hold:

  1. page.frames has exactly 2 entries (main frame + one child)
  2. page.content() contains a <frameset> tag
  3. The frameset has rows="100%", cols="100%", rows="*", cols="*", or no explicit rows/cols attribute (all equivalent to a single full-viewport frame)

This is conservative: multi-frame layouts (nav + content panes) have more than 2 frames and are never auto-followed.

Implementation Sketch

1. Add follow_frames to CrawlerRunConfig:

# async_configs.py
follow_frames: bool = True  # in __init__ signature
self.follow_frames = follow_frames  # in __init__ body
"follow_frames": self.follow_frames  # in to_dict()

2. Add a detection helper method to the crawler strategy:

async def _try_follow_frameset(self, page: Page) -> tuple:
    """
    If the page is a single full-viewport frameset, return the child frame's
    (html, url). Otherwise return (None, None).
    No extra HTTP request — Playwright has already loaded the frame.
    """
    import re
    try:
        frames = page.frames
        if len(frames) != 2:
            return None, None

        main_html = await page.content()
        if "<frameset" not in main_html.lower():
            return None, None

        match = re.search(r'<frameset([^>]*)>', main_html, re.IGNORECASE)
        if not match:
            return None, None

        attrs = match.group(1).lower()
        rows = re.search(r'rows=["\']([^"\']+)["\']', attrs)
        cols = re.search(r'cols=["\']([^"\']+)["\']', attrs)

        def is_full_viewport(val: str) -> bool:
            parts = [p.strip() for p in val.split(",")]
            return len(parts) == 1 and parts[0] in ("100%", "*")

        if rows and not is_full_viewport(rows.group(1)):
            return None, None
        if cols and not is_full_viewport(cols.group(1)):
            return None, None

        child_frame = frames[1]
        child_html = await child_frame.content()
        child_url = child_frame.url

        if not child_html or not child_url:
            return None, None

        return child_html, child_url

    except Exception:
        return None, None

3. Call it in _crawl_web after Phase 4 (overlay removal), before Phase 5 (HTML capture):

# Phase 4.5: Frameset follow
if config.follow_frames:
    frame_html, frame_url = await self._try_follow_frameset(page)
    if frame_html is not None:
        # Surface the frame URL the same way HTTP redirects are surfaced
        return AsyncCrawlResponse(
            html=frame_html,
            response_headers=response_headers,
            status_code=status_code,
            redirected_url=frame_url,
            redirected_status_code=redirected_status_code,
        )

# Phase 5: normal HTML capture
html = await page.content()
Expected behaviour after fix
result = await crawler.arun("https://www.variosolar.de/")
print(result.success)        # True
print(result.redirected_url) # "https://www.varioself.de/bauweise/varioenergy/"
print(len(result.html))      # ~28000 (full page content)

Why Not Default-On?

A full-viewport single-frame frameset has no meaningful use as raw output — the shell contains zero content by design. However, silently crawling a different URL (and potentially a different domain) than the one provided could be unexpected in domain-scoped crawls or URL validation workflows. Recommended path:

  • Ship as follow_frames=True (default on) since the frameset shell is never the desired output
  • Document follow_frames=False as the escape hatch for users who explicitly want the raw frameset response

Why Not Extend process_iframes?

<iframe> and <frameset>/<frame> are fundamentally different:

<iframe> <frameset>/<frame>
Part of the parent DOM Yes No — separate browsing context
process_iframes handles it Yes No
Deprecated in HTML5 No Yes
Typical use Embedded widgets Full-page redirect wrapper
Desired crawl behaviour Inline content into page Replace page content entirely

These warrant separate flags with separate semantics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with async_configs.py to trace CrawlerRunConfig, then follow the crawler strategy's _crawl_web flow around Phase 4 overlay removal and Phase 5 HTML capture. Verify how page.frames, frame content, and redirected_url are represented before implementing the requested single-frame behavior. Done means qualifying framesets return the child frame content and URL while non-qualifying pages and follow_frames=False retain normal capture.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
backend, web-dev
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.