[Bug]: browser_mode=builtin fails with "CDP endpoint not ready" — Chromium segfaults (SIGSEGV) when spawned from gunicorn worker (v0.9.3)

crawl4ai version

0.9.3

Expected Behavior

With browser_config.browser_mode = "builtin", the Docker server should launch its managed Chromium and return the crawled page. This worked on 0.9.1 — we use builtin mode as the only persistent-profile path that passes the server's UNTRUSTED validation (cookies/fingerprints accumulate across calls).

Current Behavior

Any POST /crawl with browser_mode="builtin" returns HTTP 500:

Exception: CDP endpoint at http://localhost:9222 is not ready after startup

ManagedBrowser.start() spawns Chromium via subprocess.Popen(args, stdout=PIPE, stderr=PIPE, preexec_fn=os.setpgrp), and the browser dies within ~500ms with SIGSEGV — logged by _initial_startup_check as:

Browser process terminated during startup | Code: -11 | STDOUT: (empty) | STDERR: (only dbus noise)

The interesting part: the exact same command line (copied from the verbose "Starting browser with args" log), same user, same env and same ulimits works fine when launched (1) via docker exec shell and (2) via a standalone python3 subprocess.Popen with identical pipes/preexec_fn — CDP answers on :9222 within ~1s in both cases. It only segfaults when the parent is the gunicorn UvicornWorker (1 worker, 4 threads, asyncio loop running). Deterministic: persists across container restarts; ~100% failure from the worker, 100% success from any other parent.

Is this reproducible?

Yes

Inputs Causing the Bug

# Any URL triggers it — only browser_mode matters
{
  "urls": ["https://example.com"],
  "browser_config": {"headless": true, "browser_mode": "builtin"},
  "crawler_config": {"cache_mode": "BYPASS"}
}

Steps to Reproduce

docker run -d --name c4ai -p 11235:11235 unclecode/crawl4ai:0.9.3
TOKEN=...  # CRAWL4AI_API_TOKEN from container env
curl -X POST localhost:11235/crawl \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"],"browser_config":{"headless":true,"browser_mode":"builtin"},"crawler_config":{"cache_mode":"BYPASS"}}'
# → 500 {"error":"CDP endpoint at http://localhost:9222 is not ready after startup"}

# Contrast test inside the same container — works:
docker exec c4ai /home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome \
  --remote-debugging-port=9222 --user-data-dir=/tmp/t --headless=new --no-sandbox \
  --disable-dev-shm-usage about:blank
# → "DevTools listening on ws://127.0.0.1:9222/..." within ~1s

Code snippets

# Standalone inside the container — this WORKS (CDP OK in 1s),
# which isolates the bug to the gunicorn-worker parent context:
import subprocess, os, time, urllib.request
args = ["/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome",
        "--remote-debugging-port=9222", "--user-data-dir=/tmp/bt", "--headless=new",
        "--window-size=1080,600", "--no-sandbox", "--disable-dev-shm-usage",
        "--disable-gpu", "about:blank"]
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=os.setpgrp)
time.sleep(1)
print(urllib.request.urlopen("http://localhost:9222/json/version", timeout=2).read())

OS

Linux x86_64 (Docker image unclecode/crawl4ai:0.9.3; host with standard Docker defaults, shm_size 4gb, ulimits nofile 65536)

Python version

3.12.14

Browser

No response

Browser version

Chromium (Playwright-installed, launched by ManagedBrowser)

Error logs & Screenshots (if applicable)

[CRAWL] api - ERROR - Crawl error: CDP endpoint at http://localhost:9222 is not ready after startup
Traceback (most recent call last):
File "/app/api.py", line 714, in handle_crawl_request
crawler = await get_crawler(browser_config)
File "/app/crawler_pool.py", line 114, in get_crawler
await crawler.start()
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 183, in start
...
Exception: CDP endpoint at http://localhost:9222 is not ready after startup

[BROWSER] Browser process terminated during startup | Code: -11 | STDOUT: | STDERR: [45969:45984:ERROR:dbus/bus.cc:405] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

Launch args used (from verbose log):

/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome --remote-debugging-port=9222 --user-data-dir=/tmp/browser-profile-XXXX --headless=new --window-size=1080,600 --no-sandbox --disable-dev-shm-usage --no-first-run --no-default-browser-check --disable-infobars --ignore-certificate-errors --disable-blink-features=AutomationControlled --disable-gpu ...

Suspects: something in the worker's process state (event-loop signal handlers?

threaded fork+exec interaction?) trips Chromium's early startup — inherited

signal dispositions or seccomp/rlimit nuance not visible in /proc//limits.

Workaround: none found short of avoiding builtin mode.

Reproduce the failure with the Docker POST request and compare it with the working docker exec and standalone subprocess cases. Start in ManagedBrowser.start() and _initial_startup_check, then trace the call from api.py handle_crawl_request through crawler_pool.py. Done means builtin mode starts Chromium from the gunicorn worker, the CDP endpoint responds, and the crawl request succeeds.