firecrawl / firecrawl/firecrawl-mcp-server
[Feature Request] Scrape timeouts and other timeout defaults configuration via environment variable
- Dominant language
- JavaScript
- Stars
- 7.5k
- Forks
- 884
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 14
Description
I have encountered very often that when using this MCP to scrape on my self hosted firecrawl, there would be instances where when using search, if a website fails to scrape due to anti bot, regardless of the search timeout, scrape will continue to run and try to scrape the website until the scrape timeout is reached. This leads to later search queries failing as often multiple pages would fail and my two firecrawl workers would not have the ability to take on more tasks, leading to later searches to timeout as well. i would like to be able to set these timeout parameters for search, scrape in search, and scrape separately via env var (assuming there is a good reason why in firecrawl self host scrape timeout doesnt inherit from search timeout) so the model doesnt have to worry about it
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating the configuration and request-handling code for search, scrape-in-search, and scrape timeouts, then determine how timeout defaults are currently set. Done means each requested timeout can be configured independently through environment variables and the behavior is covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100