allenai / allenai/ir_datasets

TREC 2023 Tip-of-the-Tongue

Aperta
#235 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
add-dataset
Lingua principale
Python
Stelle
391
Fork
58
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

**Dataset Information:**

The training and dev data of the TREC 2023 Tip-of-the-Tongue track are now available: https://trec-tot.github.io/guidelines

Description from the [website](https://trec-tot.github.io/guidelines):

```
Tip of the tongue: The phenomenon of failing to retrieve something from memory, combined with partial recall and the feeling that retrieval is imminent
```

```
In terms of input and output, the movie identification task is relatively straightforward—given an input TOT request, output a ranked list of movies. Each movie must be identified by its Wikipedia page id and the correct movie should be ranked as high as possible. For each query, runs should return a ranked list of 1000 Wikipedia page ids. Runs will be evaluated using IR metrics that are appropriate for IR tasks with one relevant document, such as discounted cumulative gain, reciprocal rank, and success@k.
```

**Dataset ID(s) & supported entities:**

- `tip-of-the-tongue/train`
- `tip-of-the-tongue/dev`
- `tip-of-the-tongue/test` (not yet released)

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.

- [x] Dataset definition (in `ir_datasets/datasets/[topid].py`)
- [x] Tests (in `tests/integration/[topid].py`)
- [x] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
- [x] Documentation (in `ir_datasets/etc/[topid].yaml`)
- [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
- [x] Downloadable content (in `ir_datasets/etc/downloads.json`)
- [ ] Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.
- [x] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.

**Additional comments/concerns/ideas/etc.**

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.