apache / apache/stormcrawler

Capture stats about hosts on status stream post fetching

Open
#504 3 comments 0 reactions 0 assignees View on GitHub
good first issue ideas
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

This could be used to notify external components of host-related information such as:
- robots delay settings
- fetch times
- [HTTP 429 codes](https://httpstatuses.com/429)

This would be used by the FetcherBolt, for instance, to implement throttling at the spout level and users would not have to modify the Fetcher code for doing so.

@owenrh

Contributor guide

Open the contributing guide

Research direction

Start by tracing the status stream and the FetcherBolt/spout interaction to determine where host information is available after fetching. Define the status data needed for robots delays, fetch times, and HTTP 429 responses, and consider how external components would consume it for throttling; done means the design and integration scope are agreed and the information is emitted consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.