allenai / allenai/ir_datasets

Add BioASQ dataset to the list of supported BEIR datasets

Abierto
#250 2 comentarios 1 reacción 0 asignados Ver en GitHub
add-dataset
Lenguaje dominante
Python
Estrellas
391
Forks
58
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi @seanmacavaney I would like to use the BioASQ dataset for an experiment and I have stumbled across this on the GitHub repo of the BEIR paper [beir-cellar](https://github.com/beir-cellar/beir/issues/86#issuecomment-1548959460) where the author links the preprocessed data for the 4 datasets marked as "unavailable". I am aware that you've been trying to extend the list of available datasets from the benchmark on ir_datasets (ie. this [issue](https://github.com/allenai/ir_datasets/issues/58)) and I was wondering if, given these resources, BioASQ could be integrated to the catalog?

**Dataset Information:**

BioASQ is a dataset featuring in the [BEIR](https://openreview.net/forum?id=wCu6T5xFjeJ) benchmark and originated from a challenge around "biomedical semantic indexing and question answering". More information about the challenge and the dataset can be found here: http://bioasq.org/

**Links to Resources:**

Link to the steps listed on [beir-cellar](https://github.com/beir-cellar/beir) in order to reproduce the files: https://github.com/beir-cellar/beir/tree/main/examples/dataset#2-bioasq ;
Link to the Google Drive space linked in the issue cited above where the preprocessed data can be found: https://drive.google.com/drive/folders/1CgDO-KmQQMpGEGeD3R20ZgTTM008xix9

**Dataset ID(s) & supported entities:**

- `beir/bioasq-2020`: queries, docs
- `beir/bioasq-2020/train`: queries, docs, qrels
- `beir/bioasq-2020/test`: queries, docs, qrels

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.

- [ ] Dataset definition (in `ir_datasets/datasets/[topid].py`)
- [ ] Tests (in `tests/integration/[topid].py`)
- [ ] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
- [ ] Documentation (in `ir_datasets/etc/[topid].yaml`)
- [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
- [ ] Downloadable content (in `ir_datasets/etc/downloads.json`)
- [ ] Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.
- [ ] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.