apache / apache/stormcrawler

fetcher.timeout.queue silently drops URLs

Open
#2,058 2 comments 0 reactions 0 assignees View on GitHub
bug core fetcher
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

When a URL waits in the queue longer than fetcher.timeout.queue, FetcherThread just logs and moves on: no status, no metric, tuple acked. The comment assumes the tuple already failed at Storm level, but that's only true if topology.message.timeout.secs is shorter — otherwise the URL vanishes and the status index never hears about it.

Emit FETCH_ERROR? At least a metric?

Noticed while planning the characterization tests for #2057 — those will pin the current behaviour for now, the fix would be its own PR.

Contributor guide

Open the contributing guide

Research direction

Start at FetcherThread's queue-timeout path and review the characterization tests planned in #2057 to understand the current behavior. Clarify whether completion should emit FETCH_ERROR, a metric, or both, then add coverage showing that a timed-out URL is reported rather than silently omitted from the status index.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.