dice-group / dice-group/Squirrel
Frontier should store timestamp of crawling
- Dominant language
- Java
- Stars
- 25
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
### Description
The Frontier should store a timestamp when it has crawled a certain URI. This timestamp could be set by the Frontier or come from the Worker. The latter would be more accurate since the worker could have crawled multiple URIs with long delays in-between.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the Frontier and Worker entry points responsible for crawling a URI and inspect how crawl state is currently stored. Clarify whether the timestamp comes from the Frontier or Worker, how it is persisted, and what tests should verify that each crawled URI records the intended time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100