LAION-AI / LAION-AI/CLIP_benchmark
add multi30k for eval
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 814
- Forks
- 103
- PR merge metrics
- No merged PRs in 30d
Description
Add the Multi30K datasets for multilingual image--sentence retrieval evaluation. The evaluation data is available in English, French, Czech, and German. The sentence data can be found on Github at https://github.com/multi30k/dataset/tree/master/data/task1/raw.
The raw untokenized sentence data can be found in the following files, where LANG = (en, cs, de, fr):
test_2016_flickr.LANG.gz
test_2017_flickr.LANG.gz
test_2018_flickr.LANG.gz
The corresponding image information can be found in https://github.com/multi30k/dataset/tree/master/data/task1/image_splits
test_2016_flickr.txt this uses the test set images from the original Flickr30K dataset.
test_2017_flickr.txt uses newly collected images
test_2018_flickr.txt uses newly collected images
The newly collected images are available to download via Google Drive. Not sure if this is easy to automatically download so re-hosting elsewhere might be possible.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Multi30K raw sentence files and image split lists linked in the issue, covering the 2016, 2017, and 2018 Flickr test sets for en, cs, de, and fr. Check how the newly collected images can be obtained or re-hosted, then verify that every specified language and split is available for multilingual image-sentence retrieval evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100