mlfoundations / mlfoundations/datacomp
Availability of npy indices for large pool
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 792
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
We're trying to produce the filtered subset of your large pool with the method "Intersection of image-based and CLIP score filtering." I know there is a script provided in the README for replicating this filtering. But the bandwidth and time required to download all the .npz embeddings is more than we have budgeted. Could you please share the ultimate .npy indices (on HF or elsewhere) so that we can soon start immediately downloading the images themselves?
I think that this .npy file would be easy for you to share (only a few GB). And I believe you must have already generated it in the past, to produce the filtered large baselines that are on the leaderboard.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the filtering-replication script described in the README and identify how the image-based and CLIP score filters produce the large pool subset. Confirm the resulting .npy indices and make them available through Hugging Face or another download location so users can fetch the images without downloading all .npz embeddings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, numpy, python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100