firecrawl / firecrawl/firecrawl

Empty catch blocks silently swallow errors across the API (redis, robots.txt, workers)

Open
#4,048 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
181k
Forks
9.8k
Avg merge
10h 19m
Merged PRs (30d)
154

Description

## Description

There are a number of empty `catch` blocks in `apps/api` that swallow failures with no logging or metrics, making degradation invisible:

- `apps/api/src/services/redis.ts` — the Redis event handlers (`error`, `reconnecting`, `connect`) wrap their own logging in `try { ... } catch (error) {}`, so a logging failure hides connection-state problems entirely.
- `apps/api/src/controllers/v1/map.ts` — `robots.txt` fetch/import failure during map is silently ignored (`catch (_) {}`), so maps may ignore robots rules without any trace.
- `apps/api/src/controllers/v0/crawl.ts` — same silent robots.txt swallow.
- `apps/api/src/lib/crawl-redis.ts` — `crawler.importRobotsTxt(sc.robots)` failures are dropped.
- `apps/api/src/services/worker/scrape-worker.ts` — a `catch (e) {}` around result handling drops errors when returning a completed job's document.
- `apps/api/src/scraper/scrapeURL/lib/smartScrape.ts` — JSON parse of an error response is swallowed (`catch (e) {}`), potentially masking cost-limit errors.

## Suggested fix

At minimum, add `logger.debug`/`logger.warn` (with context) and, where relevant, a metric to each of these sites so silent degradation becomes observable. For the robots.txt paths, a warn-level log seems appropriate since behavior changes (rules not applied) when the fetch fails.

Contributor guide

Open the contributing guide

Research direction

Review the six locations named in apps/api, starting with the Redis handlers in services/redis.ts and the robots.txt paths in controllers/v1/map.ts, controllers/v0/crawl.ts, and lib/crawl-redis.ts. Then inspect the empty catches in scrape-worker.ts and smartScrape.ts. Done means each failure is observable through appropriately contextualized debug or warn logging and, where relevant, a metric, without silently swallowing degradation.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.