JakeYallop / JakeYallop/WaybackDownloader
Allow creating additional workers for snapshot page downloading to achieve a specific download speed
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 12
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
The downloading of web pages has a variable rate limit, and new workers are created to reach to desired limit. Downloading lists of CDX records (snapshot pages) can take several seconds, and with a high-enough webpage download rate limit, it could end up making the snapshot page downloading the limiting factor.
We should try to rewrite the snapshot downloading so that it can also be parallelised.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the snapshot-page download path, including CDX record retrieval, worker creation, and rate limiting. Determine how additional workers are currently created for webpage downloads, then define completion as snapshot-page downloading being parallelized without preventing the configured overall download speed from being reached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100