Send host info on a specific stream
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
Could be used to notify that a status HTTP 429 has been caught (see ##784 on that topic) or that the robots.txt specified a delay.
A bespoke bolt could then notify a URL Frontier via [setDelay](https://github.com/crawler-commons/url-frontier/blob/master/API/urlfrontier.proto#L55) or [blockQueueUntil](https://github.com/crawler-commons/url-frontier/blob/master/API/urlfrontier.proto#L45)
see #504 for a related discussion
Contributor guide
Research direction
Start with the issue's references to #784 and #504, then review the URL Frontier API links for setDelay and blockQueueUntil. The proposed stream-specific host information and bespoke bolt need a defined design and integration path. Done would require an agreed behavior and implementation scope, but the issue does not provide acceptance criteria or named project files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100