unclecode / unclecode/crawl4ai

[Bug]: Playground Advanced Config panel is non-functional — still sends the removed `code` protocol to /config/dump

Open
#2,260 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⚙ Done 🐞 Bug 🐳 Docker 📌 Root caused
Dominant language
Python
Stars
83.9k
Forks
8.7k
Avg merge
3d 7h
Merged PRs (30d)
11

Description

Follow-up to #2222 / #2224. That PR stopped md/llm from being collateral damage. The underlying problem is still here, and it now fails more quietly than before.

What's wrong

The playground's Advanced Config panel still speaks the {type, code} protocol that /config/dump dropped in the 0.8.x security work. code is a globally forbidden field under the untrusted trust boundary, so the pre-flight always 400s:

CrawlerRunConfig.load({"type": "CrawlerRunConfig", "code": "..."}, provenance=Provenance.UNTRUSTED)
→ UntrustedConfigError: field 'code' is not permitted on CrawlerRunConfig from an untrusted request

crawl and crawl_stream survive only because of the regex fallback in runCrawl() (deploy/docker/static/playground/index.html), which checks the editor text for stream=True. Two consequences:

  1. Config is silently discarded. With the default snippet CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS), the fallback sends {crawler_config: {type: 'CrawlerRunConfig', params: {stream: true}}}. The cache_mode line never reaches the server. The run succeeds, so nothing tells the user their config was dropped. Anything they type beyond stream is ignored.
  2. BrowserConfig is a hard failure. That template has no stream=True, so no fallback fires and the run aborts with ✖ config error.

So the panel is non-functional on the only endpoint that still displays it.

Why it can't be patched in place

/config/dump used to eval the snippet. That was removed on purpose — it was a gadget-construction oracle. A Python-snippet editor can't work against the current server, so this isn't a matter of fixing the request shape.

Options
  • Replace the CodeMirror Python editor with a params/JSON editor that posts {type, params}, which is what /config/dump accepts today. Keeps the feature, matches the server.
  • Remove the panel. /crawl is still fully usable through the API; the playground just stops advertising a capability it doesn't have.

Either way the regex fallback in runCrawl() should go — it exists only to paper over the broken pre-flight, and it's what makes the config loss silent.

Repro
  1. Start the Docker server, open /playground/
  2. Endpoint crawl, default snippet, Run → succeeds, but cache_mode was never sent
  3. Switch Type to BrowserConfig, Run → ✖ config error, run aborts
Notes
  • Minor, same area: #cfg-status is written only by pyConfigToJson(). Since #2224, md/llm skip that call, so a stale ✖ config error from a previous crawl run stays on screen.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in deploy/docker/static/playground/index.html, especially runCrawl() and pyConfigToJson(), then reproduce the crawl and BrowserConfig cases against /config/dump. Trace the current {type, code} request and the server's accepted {type, params} shape before choosing whether to replace or remove the panel. Done means the panel no longer relies on the removed protocol or regex fallback, and config status cannot remain stale.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
api, frontend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.