page_timeout is silently capped at 60s over HTTP with no way to raise it

Open

#2,211 opened on Aug 28, 2026 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⚙️ In-progress 🐞 Bug 📌 Root caused

Dominant language: Python
Stars: 83.9k
Forks: 8.7k
Avg merge: 3d 7h
Merged PRs (30d): 11

Description

Problem

_clamp_untrusted caps page_timeout and wait_for_timeout at _MAX_TIMEOUT_MS = 60_000 for any config arriving over HTTP. The Docker server passes Provenance.UNTRUSTED at every /crawl call site, so a client asking for more is silently given 60s — no warning, no field in the response. A page that legitimately takes longer always fails with Page.goto: Timeout 60000ms exceeded, quoting a number the caller never sent.

The cap itself is reasonable for a public server. Not being able to change it is the problem: it isn't in config.yml, no env var overrides it, and the value is a module-level literal.

crawler.base_config looks like the intended escape hatch and isn't — it's applied post-deserialization with setattr, genuinely bypassing the clamp, but only for fields that are None or ''. page_timeout always carries its 60000 default, so setting it there does nothing, silently.

Expected

An operator running their own container can raise the ceiling — an env var, a limits: entry alongside wall_clock_s, or a base_config that applies to defaults too. Failing that, a clamped request should say so rather than report a timeout the caller didn't ask for.

Reproduce (docker, 0.9.2)

A server that answers after 90s:

import time
from http.server import BaseHTTPRequestHandler, HTTPServer
BODY = b"<html><body><h1>Slow</h1><p>" + b"x"*600 + b"</p></body></html>"
class H(BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(90)
        self.send_response(200); self.send_header("Content-Length", str(len(BODY))); self.end_headers()
        self.wfile.write(BODY)
HTTPServer(("0.0.0.0", 8080), H).serve_forever()

docker network create repro
docker run -d --name slowsrv --network repro -v $PWD/slow_server.py:/srv/s.py:ro python:3.12-slim python /srv/s.py
docker run -d --name c4ai --network repro -p 11235:11235 --shm-size=2gb \
  -e CRAWL4AI_API_TOKEN=dev-token -e CRAWL4AI_ALLOW_INTERNAL_URLS=true unclecode/crawl4ai:0.9.2

curl -X POST http://localhost:11235/crawl -H 'authorization: Bearer dev-token' \
  -H 'content-type: application/json' \
  -d '{"urls":["http://slowsrv:8080/"],"crawler_config":{"page_timeout":300000,"wait_until":"domcontentloaded"}}'

Before — asked for 300s:

HTTP 500 in 60.7s
Page.goto: Timeout 60000ms exceeded

After — same request, against an image built with _MAX_TIMEOUT_MS = 600_000:

HTTP 200 in 90.9s
success: true, markdown: "# Slow page\nxxxx…"

Suggested patch

Make the ceiling configurable rather than removing it:

_MAX_TIMEOUT_MS = int(os.environ.get("CRAWL4AI_MAX_TIMEOUT_MS", 60_000))

An env var keeps the safe default for a public deployment while letting an operator raise it for a trusted one. A limits.max_timeout_ms key in config.yml, next to wall_clock_s, would fit the existing "0 = unbounded" convention equally well.

Separately, base_config skipping any field with a non-empty default is surprising enough to be worth documenting.

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start at _clamp_untrusted and _MAX_TIMEOUT_MS, then trace the Docker server’s /crawl call sites and the post-deserialization base_config handling. Compare the existing config.yml settings and environment configuration paths before choosing how an operator can raise the ceiling. Done means trusted deployments can configure the maximum without weakening the default cap, and clamped requests no longer misleadingly report an unrequested timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack: docker, python
Domain: api, backend, devops
Issue type: Feature
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Active
Clarity: Mostly clear
Newbie friendliness: 58/100