unclecode / unclecode/crawl4ai
[Bug]: LLMExtractionStrategy not applied when CrawlerRunConfig.cache_mode=ENABLED
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 83.9k
- Forks
- 8.7k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 11
Description
crawl4ai version
0.7.4
Expected Behavior
When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is filled with freshly generated output of LLM.
Current Behavior
When using cache_mode=CacheMode.ENABLED on CrawlerRunConfig and a previously crawled and cached URL the result.extracted_content is empty.
Is this reproducible?
Yes
Code snippets
import os
import asyncio
import json
from pydantic import BaseModel, Field
from typing import List
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
class Product(BaseModel):
name: str
price: str
async def main():
# 1. Define the LLM extraction strategy
llm_strategy = LLMExtractionStrategy(
llm_config = LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv('OPENAI_API_KEY')),
schema=Product.schema_json(), # Or use model_json_schema()
extraction_type="schema",
instruction="Extract all product objects with 'name' and 'price' from the content.",
chunk_token_threshold=1000,
overlap_rate=0.0,
apply_chunking=True,
input_format="markdown", # or "html", "fit_markdown"
extra_args={"temperature": 0.0, "max_tokens": 800}
)
# 2. Build the crawler config
crawl_config = CrawlerRunConfig(
extraction_strategy=llm_strategy,
cache_mode=CacheMode.ENABLED # (!!) BYPASS is working
)
# 3. Create a browser config if needed
browser_cfg = BrowserConfig(headless=True)
async with AsyncWebCrawler(config=browser_cfg) as crawler:
# 4. Let's say we want to crawl a single page
result = await crawler.arun(
url="https://example.com/products",
config=crawl_config
)
if result.success:
# 5. The extracted content is presumably JSON
data = json.loads(result.extracted_content)
print("Extracted items:", data)
# 6. Show usage stats
llm_strategy.show_usage() # prints token usage
else:
print("Error:", result.error_message)
The Code Example is taken from the LLM Strategies Documentation
OS
macOS
Python version
3.11.4
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied AsyncWebCrawler.arun example and compare CacheMode.ENABLED with the working BYPASS case. Trace how cached results are handled alongside LLMExtractionStrategy, then verify that a cached URL still produces populated result.extracted_content and that the documented example works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100