formulabun / formulabun/scraper
asynchronize the server scraping
Open
- Dominant language
- Go
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Currently scraping the files from the ms goes in order of the servers, and then file per file. This is just slow.
It's better to:
- collect all (files,checksum)
- sort the servers hosting those based on distance per file, ie. get the best download server for a file
- maybe do something with the sizes of the files
- asynchronously download files with a given number of work routines
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.