unclecode / unclecode/crawl4ai

[Bug]: LLMExtractionStrategy not applied when CrawlerRunConfig.cache_mode=ENABLED

Open
#1,455 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

✊🏼 On-hold 🐞 Bug 📌 Root caused
Dominant language
Python
Stars
83.9k
Forks
8.7k
Avg merge
3d 7h
Merged PRs (30d)
11

Description

crawl4ai version

0.7.4

Expected Behavior

When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is filled with freshly generated output of LLM.

Current Behavior

When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is empty.

Is this reproducible?

Yes

Code snippets
import os
import asyncio
import json
from pydantic import BaseModel, Field
from typing import List
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Product(BaseModel):
    name: str
    price: str

async def main():
    # 1. Define the LLM extraction strategy
    llm_strategy = LLMExtractionStrategy(
        llm_config = LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv('OPENAI_API_KEY')),
        schema=Product.schema_json(), # Or use model_json_schema()
        extraction_type="schema",
        instruction="Extract all product objects with 'name' and 'price' from the content.",
        chunk_token_threshold=1000,
        overlap_rate=0.0,
        apply_chunking=True,
        input_format="markdown",   # or "html", "fit_markdown"
        extra_args={"temperature": 0.0, "max_tokens": 800}
    )

    # 2. Build the crawler config
    crawl_config = CrawlerRunConfig(
        extraction_strategy=llm_strategy,
        cache_mode=CacheMode.ENABLED # (!!) BYPASS is working
    )

    # 3. Create a browser config if needed
    browser_cfg = BrowserConfig(headless=True)

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        # 4. Let's say we want to crawl a single page
        result = await crawler.arun(
            url="https://example.com/products",
            config=crawl_config
        )

        if result.success:
            # 5. The extracted content is presumably JSON
            data = json.loads(result.extracted_content)
            print("Extracted items:", data)

            # 6. Show usage stats
            llm_strategy.show_usage()  # prints token usage
        else:
            print("Error:", result.error_message)

The Code Example is taken from the LLM Strategies Documentation

OS

macOS

Python version

3.11.4

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied AsyncWebCrawler.arun example and compare CacheMode.ENABLED with the working BYPASS case. Trace how cached results are handled alongside LLMExtractionStrategy, then verify that a cached URL still produces populated result.extracted_content and that the documented example works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.