allenai / allenai/ir_datasets

TREC 2023 Tip-of-the-Tongue

Đang mở
#235 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
add-dataset
Ngôn ngữ chính
Python
Star
391
Fork
58
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

**Dataset Information:**

The training and dev data of the TREC 2023 Tip-of-the-Tongue track are now available: https://trec-tot.github.io/guidelines

Description from the [website](https://trec-tot.github.io/guidelines):

```
Tip of the tongue: The phenomenon of failing to retrieve something from memory, combined with partial recall and the feeling that retrieval is imminent
```

```
In terms of input and output, the movie identification task is relatively straightforward—given an input TOT request, output a ranked list of movies. Each movie must be identified by its Wikipedia page id and the correct movie should be ranked as high as possible. For each query, runs should return a ranked list of 1000 Wikipedia page ids. Runs will be evaluated using IR metrics that are appropriate for IR tasks with one relevant document, such as discounted cumulative gain, reciprocal rank, and success@k.
```

**Dataset ID(s) & supported entities:**

- `tip-of-the-tongue/train`
- `tip-of-the-tongue/dev`
- `tip-of-the-tongue/test` (not yet released)

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.

- [x] Dataset definition (in `ir_datasets/datasets/[topid].py`)
- [x] Tests (in `tests/integration/[topid].py`)
- [x] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
- [x] Documentation (in `ir_datasets/etc/[topid].yaml`)
- [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
- [x] Downloadable content (in `ir_datasets/etc/downloads.json`)
- [ ] Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.
- [x] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.

**Additional comments/concerns/ideas/etc.**

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.