internetarchive / internetarchive/dweb-mirror
Crawling fails - leaving tasks
Open
2SC/ON
- Dominant language
- JavaScript
- Stars
- 330
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Crawling fails on two of the the edge cases in https://github.com/internetarchive/dweb-archive/issues/120. In both cases, presence in the crawl causes left over tasks
- [ ] bdc-W3PD1123
- [ ] JournetsinPersiaAndKurdistanVolII
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the related issue 120 and reproducing crawls involving bdc-W3PD1123 and JournetsinPersiaAndKurdistanVolII. Trace why each crawl leaves a task behind and compare the crawler's behavior for these two edge cases. Done means both crawls complete without leftover tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100