internetarchive / internetarchive/dweb-mirror
Crawl and limitTotalTasks
- Dominant language
- JavaScript
- Stars
- 330
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Either crawl is ignoring limitTotalTasks *or* its saying it completed when it didn't .
To test .... try a large crawl (like bali-lontar-transcribed) which is ~12k tasks
It will report that it got all of them, and appears to have done so, even though default limit is 3k tasks.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the crawl entry point and reproduce the issue with the large bali-lontar-transcribed crawl, which is described as roughly 12k tasks. Compare the reported completion count with the default limitTotalTasks value of 3k; done means the crawl respects the limit and accurately reports how many tasks were completed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100