fetcher.timeout.queue silently drops URLs
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
When a URL waits in the queue longer than fetcher.timeout.queue, FetcherThread just logs and moves on: no status, no metric, tuple acked. The comment assumes the tuple already failed at Storm level, but that's only true if topology.message.timeout.secs is shorter — otherwise the URL vanishes and the status index never hears about it.
Emit FETCH_ERROR? At least a metric?
Noticed while planning the characterization tests for #2057 — those will pin the current behaviour for now, the fix would be its own PR.
Contributor guide
Research direction
Start at FetcherThread's queue-timeout path and review the characterization tests planned in #2057 to understand the current behavior. Clarify whether completion should emit FETCH_ERROR, a metric, or both, then add coverage showing that a timed-out URL is reported rather than silently omitted from the status index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100