download datasets using bittorrent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
🚀 The feature, motivation and pitch
I'm the sysadmin of one of the servers that hosts datasets that appear on torch.datasets (DTD, FGVCAircraft, Flowers102, and OxfordIIITPet) . We're happy that our stuff is so widely used but that's also taking a hit on other stuff on the server and the downloads sometimes take a very long time.
We've prepared torrents for some of our datasets, uploaded them to academic torrents, and recommend to use it over http. I was wondering if it was possible to add the option to download datasets over bittorrent.
Alternatives
No response
Additional context
No response
cc @pmeier @YosuaMichael
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing how the DTD, FGVCAircraft, Flowers102, and OxfordIIITPet datasets are downloaded through torch.datasets, then assess the proposed Academic Torrents integration. Done should mean there is a decided, supported way to offer BitTorrent downloads without disrupting existing HTTP downloads, with coverage for the affected datasets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100