unclecode / unclecode/crawl4ai
[Bug]: Docker pool's permanent browser never serves any request - signature is computed without the egress proxy, so no `/crawl` can ever match it
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 83.9k
- Forks
- 8.7k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 11
Description
Summary
Found while investigating #2202. The Docker server's "permanent" warm browser is dead weight: it is started at boot, serves zero requests, is never cleaned up, and causes every container to run a second browser tree from the first request onward.
Root cause
The pool matches requests to browsers by hashing the full BrowserConfig (crawler_pool.py:46-49), and proxy_config is part of to_dict() (async_configs.py:968).
init_permanent()is called at startup with a config built straight fromconfig.yml— withoutenforce_egress()(server.py:199-203), so its fingerprint hasproxy_config: None.- Every
/crawlrequest's config goes throughenforce_egress()first (api.py:687-690), which setsproxy_configto the egress pinning proxy — whose port is random each boot (egress_broker.py:198-201).
The two fingerprints can never be equal, so _is_default_config() (crawler_pool.py:60) never matches and PERMANENT is never returned. Note get_default_browser_config() (server.py:126-141) does apply enforce_egress, so even the server's own /html, /screenshot, /pdf, /execute_js endpoints miss it.
Additionally, the janitor only sweeps HOT_POOL and COLD_POOL (crawler_pool.py:177,198) — PERMANENT is never inspected, so the unused browser also can never be reclaimed.
Impact
- ~270 MB RSS + one Playwright driver + full Chromium tree per container, doing nothing, forever.
- Every container shows two
driver → browserprocess branches after the first request (observed in #2202's process listing: one branch from boot, one from the first crawl a day later). - The
text_modeoptimization configured for the default browser never applies to real traffic.
Verified
Reproduced on unclecode/crawl4ai:0.9.2: fresh container = one browser tree; after a single plain /crawl = two independent trees, permanent one idle.
Proposed fix
Keep the permanent browser, fix the match: build its config through the same path requests use (get_default_browser_config(), i.e. after enforce_egress), and compute DEFAULT_CONFIG_SIG from that. Alternatively/additionally, exclude the server-injected proxy_config from the pool signature, since the server sets it identically on every request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crawler_pool.py:46-49, 60, 177, and 198, then trace the permanent-browser setup in server.py:126-141 and 199-203 and request handling in api.py:687-690. Verify that the permanent browser uses a matching request signature, serves the default traffic, and is included in janitor cleanup without creating a second browser tree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100