clearlydefined / clearlydefined/crawler

No timeout set up on fetch calls

Open
#760 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

## Problem

All outbound HTTP requests use `axios` with no `timeout` configured (defaults to `0` / no limit). A single unresponsive upstream can block a crawler worker indefinitely. `axios-retry` only triggers on errors — a hanging connection never retries.

Affects all fetch providers: npm, Maven, NuGet, PyPI, crates.io, Conda, Debian, Go, Gradle Plugin, Packagist, CocoaPods, RubyGems, and `getStream` tarball downloads.

## Proposed fix

Add a default `timeout` to the axios instances in `lib/fetch.js`:
- ~30s for metadata/API calls (`callFetch`, `callFetchWithRetry`)
- ~300s for large downloads (`getStream`)
- Allow callers to override per-request

Timed-out requests will be treated as network errors and retried via existing `axios-retry` logic.

### Files affected
- `lib/fetch.js` — core axios instances and request builders
- `providers/fetch/goFetch.js` — separate axios instance also missing timeout

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.