apache / apache/stormcrawler

Send host info on a specific stream

Closed
#867 3 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

Could be used to notify that a status HTTP 429 has been caught (see ##784 on that topic) or that the robots.txt specified a delay.
A bespoke bolt could then notify a URL Frontier via [setDelay](https://github.com/crawler-commons/url-frontier/blob/master/API/urlfrontier.proto#L55) or [blockQueueUntil](https://github.com/crawler-commons/url-frontier/blob/master/API/urlfrontier.proto#L45)

see #504 for a related discussion

Contributor guide

Open the contributing guide

Research direction

Start with the issue's references to #784 and #504, then review the URL Frontier API links for setDelay and blockQueueUntil. The proposed stream-specific host information and bespoke bolt need a defined design and integration path. Done would require an agreed behavior and implementation scope, but the issue does not provide acceptance criteria or named project files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.