dice-group / dice-group/Squirrel

Frontier should store timestamp of crawling

Open
#27 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
25
Forks
19
PR merge metrics
No merged PRs in 30d

Description

### Description

The Frontier should store a timestamp when it has crawled a certain URI. This timestamp could be set by the Frontier or come from the Worker. The latter would be more accurate since the worker could have crawled multiple URIs with long delays in-between.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the Frontier and Worker entry points responsible for crawling a URI and inspect how crawl state is currently stored. Clarify whether the timestamp comes from the Frontier or Worker, how it is persisted, and what tests should verify that each crawled URI records the intended time.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.