LAION-AI / LAION-AI/CLIP_benchmark

add multi30k for eval

Open
#46 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

new datasets
Dominant language
Python
Stars
814
Forks
103
PR merge metrics
No merged PRs in 30d

Description

Add the Multi30K datasets for multilingual image--sentence retrieval evaluation. The evaluation data is available in English, French, Czech, and German. The sentence data can be found on Github at https://github.com/multi30k/dataset/tree/master/data/task1/raw.

The raw untokenized sentence data can be found in the following files, where LANG = (en, cs, de, fr):

test_2016_flickr.LANG.gz
test_2017_flickr.LANG.gz
test_2018_flickr.LANG.gz

The corresponding image information can be found in https://github.com/multi30k/dataset/tree/master/data/task1/image_splits

test_2016_flickr.txt this uses the test set images from the original Flickr30K dataset.
test_2017_flickr.txt uses newly collected images
test_2018_flickr.txt uses newly collected images

The newly collected images are available to download via Google Drive. Not sure if this is easy to automatically download so re-hosting elsewhere might be possible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Multi30K raw sentence files and image split lists linked in the issue, covering the 2016, 2017, and 2018 Flickr test sets for en, cs, de, and fr. Check how the newly collected images can be obtained or re-hosted, then verify that every specified language and split is available for multilingual image-sentence retrieval evaluation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.