firecrawl / firecrawl/firecrawl-mcp-server

[Feature Request] Scrape timeouts and other timeout defaults configuration via environment variable

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.5k
Forks
884
Avg merge
1d 11h
Merged PRs (30d)
14

Description

I have encountered very often that when using this MCP to scrape on my self hosted firecrawl, there would be instances where when using search, if a website fails to scrape due to anti bot, regardless of the search timeout, scrape will continue to run and try to scrape the website until the scrape timeout is reached. This leads to later search queries failing as often multiple pages would fail and my two firecrawl workers would not have the ability to take on more tasks, leading to later searches to timeout as well. i would like to be able to set these timeout parameters for search, scrape in search, and scrape separately via env var (assuming there is a good reason why in firecrawl self host scrape timeout doesnt inherit from search timeout) so the model doesnt have to worry about it

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the configuration and request-handling code for search, scrape-in-search, and scrape timeouts, then determine how timeout defaults are currently set. Done means each requested timeout can be configured independently through environment variables and the behavior is covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.