unclecode / unclecode/crawl4ai
[Bug]: crawler.base_config boolean values are silently ignored (regression from #1505)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 83.9k
- Forks
- 8.7k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 11
Description
crawl4ai version
0.9.2
Expected Behavior
config.yml sets a server-side default for every crawl:
# deploy/docker/config.yml:74-76
crawler:
base_config:
simulate_user: true
A POST /crawl request that does not send simulate_user should run with simulate_user=True, i.e. the server default applies.
A request that does send simulate_user should win over the server default — that is the intent of #1505.
Current Behavior
The server default never applies. simulate_user is False on every crawl.
api.py:715 reads "the client didn't send this field" as "the attribute is None or """:
current_value = getattr(crawler_config, key)
if current_value is None or current_value == "": # api.py:715
setattr(crawler_config, key, value)
CrawlerRunConfig.simulate_user defaults to False (async_configs.py:1650), and False is neither None nor "", so the guard never passes.
The same goes for every base_config key defaulting to a boolean or a number: magic, override_navigator, check_robots_txt, remove_overlay_elements, page_timeout.
The config-list path at api.py:707 (8995c1b, #1837) copies the guard.
So the stock image ships simulate_user: true (config.yml:74-76, utils.py:63)
but never injects the navigator_overrider script (browser_manager.py:1229-1235) or runs the mouse-move simulation (async_crawler_strategy.py:980-983).
a1950af (#1505) introduced this.
The setattr used to be unconditional and clobbered client-sent values, so reverting brings #1505 back.
After CrawlerRunConfig.load() (api.py:675) nothing tells "omitted" apart from "sent, equal to the default".
That information only exists in the raw request dict
Is this reproducible?
Yes
Inputs Causing the Bug
- URL(s): any, e.g. https://example.com
- Settings used: stock deploy/docker/config.yml, i.e. crawler.base_config.simulate_user: true
- Input data: {"urls": ["https://example.com"]} # no crawler_config key
Steps to Reproduce
1. Start the stock server image, config.yml untouched.
2. POST the body above to /crawl.
3. Read effective_config in handle_crawl_request (api.py:697-716).
simulate_user is False.
Code snippets
# The guard in isolation. No server or browser needed.
from crawl4ai import CrawlerRunConfig
cfg = CrawlerRunConfig() # client sent no crawler_config
value = getattr(cfg, "simulate_user") # False, the dataclass default
assert value is None or value == "" # api.py:715 -> fails, setattr skipped
OS
Linux (Docker image, python:3.12-slim-bookworm)
Python version
3.12
Browser
Chromium (Playwright, headless)
Browser version
No response
Error logs & Screenshots (if applicable)
No error. The server drops the value silently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in api.py around CrawlerRunConfig.load() at line 675 and effective_config handling at lines 697-716; compare the normal and config-list paths, including the guard at line 715. Use the raw request dictionary to distinguish omitted fields from explicitly sent defaults. Done means base_config boolean and numeric defaults apply when omitted while explicit client values remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100