dice-group / dice-group/Squirrel

Improvement of the HTTP Fetcher and Queue when status 500/503 is returned

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
component:worker priority:low type:enhancement
Dominant language
Java
Stars
25
Forks
19
PR merge metrics
No merged PRs in 30d

Description

When the Fetcher tries to access an URI and gets status 500/503 is returned, the URI is removed when the response (empty list of triples) is sent to the Frontier.
The Fetcher should implement the Retry Pattern. A parameter should be passed to the frontier to determine
how many retries should execute and the waiting time between retries. If still, a flag should be put in Crawleable URI data map, so the queue won't delete this URI and may crawl it later.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Fetcher path for HTTP 500/503 responses through the Frontier and the queue's Crawlable URI data map. Confirm how empty triple responses currently remove URIs, then define completion as configurable retry counts and wait times, with failed URIs retained for later crawling after retries are exhausted.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.