Capture stats about hosts on status stream post fetching
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
This could be used to notify external components of host-related information such as:
- robots delay settings
- fetch times
- [HTTP 429 codes](https://httpstatuses.com/429)
This would be used by the FetcherBolt, for instance, to implement throttling at the spout level and users would not have to modify the Fetcher code for doing so.
@owenrh
Contributor guide
Research direction
Start by tracing the status stream and the FetcherBolt/spout interaction to determine where host information is available after fetching. Define the status data needed for robots delays, fetch times, and HTTP 429 responses, and consider how external components would consume it for throttling; done means the design and integration scope are agreed and the information is emitted consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100