unclecode / unclecode/crawl4ai
[Bug]: Playground Advanced Config panel is non-functional — still sends the removed `code` protocol to /config/dump
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 83.9k
- Forks
- 8.7k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 11
Description
Follow-up to #2222 / #2224. That PR stopped md/llm from being collateral damage. The underlying problem is still here, and it now fails more quietly than before.
What's wrong
The playground's Advanced Config panel still speaks the {type, code} protocol that /config/dump dropped in the 0.8.x security work. code is a globally forbidden field under the untrusted trust boundary, so the pre-flight always 400s:
CrawlerRunConfig.load({"type": "CrawlerRunConfig", "code": "..."}, provenance=Provenance.UNTRUSTED)
→ UntrustedConfigError: field 'code' is not permitted on CrawlerRunConfig from an untrusted request
crawl and crawl_stream survive only because of the regex fallback in runCrawl() (deploy/docker/static/playground/index.html), which checks the editor text for stream=True. Two consequences:
- Config is silently discarded. With the default snippet
CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS), the fallback sends{crawler_config: {type: 'CrawlerRunConfig', params: {stream: true}}}. Thecache_modeline never reaches the server. The run succeeds, so nothing tells the user their config was dropped. Anything they type beyondstreamis ignored. - BrowserConfig is a hard failure. That template has no
stream=True, so no fallback fires and the run aborts with✖ config error.
So the panel is non-functional on the only endpoint that still displays it.
Why it can't be patched in place
/config/dump used to eval the snippet. That was removed on purpose — it was a gadget-construction oracle. A Python-snippet editor can't work against the current server, so this isn't a matter of fixing the request shape.
Options
- Replace the CodeMirror Python editor with a params/JSON editor that posts
{type, params}, which is what/config/dumpaccepts today. Keeps the feature, matches the server. - Remove the panel.
/crawlis still fully usable through the API; the playground just stops advertising a capability it doesn't have.
Either way the regex fallback in runCrawl() should go — it exists only to paper over the broken pre-flight, and it's what makes the config loss silent.
Repro
- Start the Docker server, open
/playground/ - Endpoint
crawl, default snippet, Run → succeeds, butcache_modewas never sent - Switch Type to
BrowserConfig, Run →✖ config error, run aborts
Notes
- Minor, same area:
#cfg-statusis written only bypyConfigToJson(). Since #2224,md/llmskip that call, so a stale✖ config errorfrom a previouscrawlrun stays on screen.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in deploy/docker/static/playground/index.html, especially runCrawl() and pyConfigToJson(), then reproduce the crawl and BrowserConfig cases against /config/dump. Trace the current {type, code} request and the server's accepted {type, params} shape before choosing whether to replace or remove the panel. Done means the panel no longer relies on the removed protocol or regex fallback, and config status cannot remain stale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, python
- Domain
- api, frontend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100