internetarchive / internetarchive/dweb-mirror

Crawl and limitTotalTasks

Open
#314 0 comments 0 reactions 0 assignees View on GitHub
2SC/ON
Dominant language
JavaScript
Stars
330
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Either crawl is ignoring limitTotalTasks *or* its saying it completed when it didn't .

To test .... try a large crawl (like bali-lontar-transcribed) which is ~12k tasks
It will report that it got all of them, and appears to have done so, even though default limit is 3k tasks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the crawl entry point and reproduce the issue with the large bali-lontar-transcribed crawl, which is described as roughly 12k tasks. Compare the reported completion count with the default limitTotalTasks value of 3k; done means the crawl respects the limit and accurately reports how many tasks were completed.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.