mlfoundations / mlfoundations/datacomp
Downloading DataComp-1B
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 792
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to download datacomp-1b, but it seems both my downloading speed and success rate are low, even after I set num_retries to 3:
worker - success: 0.887 - failed to download: 0.101 - failed to resize: 0.012 - images per sec: 8 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 103 - count: 247671
worker - success: 0.889 - failed to download: 0.101 - failed to resize: 0.010 - images per sec: 9 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 107 - count: 257671
worker - success: 0.894 - failed to download: 0.098 - failed to resize: 0.008 - images per sec: 8 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 111 - count: 267671
29it [40:28, 7.45s/it]worker - success: 0.890 - failed to download: 0.100 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 115 - count: 277671
worker - success: 0.887 - failed to download: 0.103 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 119 - count: 287671
worker - success: 0.891 - failed to download: 0.099 - failed to resize: 0.009 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 123 - count: 297671
worker - success: 0.882 - failed to download: 0.108 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 127 - count: 307671
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names DataComp-1B and num_retries=3 but no files or tests. Start by reproducing the download and examining the reported download, resize, success-rate, and throughput metrics; clarify the expected behavior and affected entry point before defining a fix. Done requires an agreed resolution for the low speed and failure rate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100