clearlydefined / clearlydefined/crawler
No timeout set up on fetch calls
- Dominant language
- JavaScript
- Stars
- 60
- Forks
- 38
- Avg merge
- 1h 49m
- Merged PRs (30d)
- 1
Description
## Problem
All outbound HTTP requests use `axios` with no `timeout` configured (defaults to `0` / no limit). A single unresponsive upstream can block a crawler worker indefinitely. `axios-retry` only triggers on errors — a hanging connection never retries.
Affects all fetch providers: npm, Maven, NuGet, PyPI, crates.io, Conda, Debian, Go, Gradle Plugin, Packagist, CocoaPods, RubyGems, and `getStream` tarball downloads.
## Proposed fix
Add a default `timeout` to the axios instances in `lib/fetch.js`:
- ~30s for metadata/API calls (`callFetch`, `callFetchWithRetry`)
- ~300s for large downloads (`getStream`)
- Allow callers to override per-request
Timed-out requests will be treated as network errors and retried via existing `axios-retry` logic.
### Files affected
- `lib/fetch.js` — core axios instances and request builders
- `providers/fetch/goFetch.js` — separate axios instance also missing timeout
Contributor guide
Assessment
This issue has not been assessed yet.