allenai / allenai/ir_datasets

Add BioASQ dataset to the list of supported BEIR datasets

Đang mở
#250 2 bình luận 1 reaction 0 người được giao Xem trên GitHub
add-dataset
Ngôn ngữ chính
Python
Star
391
Fork
58
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

Hi @seanmacavaney I would like to use the BioASQ dataset for an experiment and I have stumbled across this on the GitHub repo of the BEIR paper [beir-cellar](https://github.com/beir-cellar/beir/issues/86#issuecomment-1548959460) where the author links the preprocessed data for the 4 datasets marked as "unavailable". I am aware that you've been trying to extend the list of available datasets from the benchmark on ir_datasets (ie. this [issue](https://github.com/allenai/ir_datasets/issues/58)) and I was wondering if, given these resources, BioASQ could be integrated to the catalog?

**Dataset Information:**

BioASQ is a dataset featuring in the [BEIR](https://openreview.net/forum?id=wCu6T5xFjeJ) benchmark and originated from a challenge around "biomedical semantic indexing and question answering". More information about the challenge and the dataset can be found here: http://bioasq.org/

**Links to Resources:**

Link to the steps listed on [beir-cellar](https://github.com/beir-cellar/beir) in order to reproduce the files: https://github.com/beir-cellar/beir/tree/main/examples/dataset#2-bioasq ;
Link to the Google Drive space linked in the issue cited above where the preprocessed data can be found: https://drive.google.com/drive/folders/1CgDO-KmQQMpGEGeD3R20ZgTTM008xix9

**Dataset ID(s) & supported entities:**

- `beir/bioasq-2020`: queries, docs
- `beir/bioasq-2020/train`: queries, docs, qrels
- `beir/bioasq-2020/test`: queries, docs, qrels

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.

- [ ] Dataset definition (in `ir_datasets/datasets/[topid].py`)
- [ ] Tests (in `tests/integration/[topid].py`)
- [ ] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
- [ ] Documentation (in `ir_datasets/etc/[topid].yaml`)
- [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
- [ ] Downloadable content (in `ir_datasets/etc/downloads.json`)
- [ ] Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.
- [ ] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.