bigscience-workshop / bigscience-workshop/biomedical

All local datasets need a list of filenames for downloading

Open
#712 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
505
Forks
117
PR merge metrics
No merged PRs in 30d

Description

All local datasets need a list of the exact filenames to download. Several files don't have any indication of what a user needs to manually download. This requires slogging through the code to find out what is needed.
For example, consider [bioasq_task_c_2017](https://github.com/bigscience-workshop/biomedical/tree/master/bigbio/biodatasets/bioasq_task_c_2017) which requires some combination of
```
BioASQ_2017_Task5C_Final.tar.gz
BioASQ_2017_Task5C_Training.tar.gz
taskCTrainingData2017.json
taskc_golden2.json
```
and expects that the `tar.gz` files are all manually uncompressed before loading. None of these details are documented in the current script. Moreover, these are the types of requirements we should automate in the script directly.

I'd propose we require either:
- A list of the exact URLs to fetch (assuming the user is authenticated as needed)
- A list of the file names themselves

We should *always* operate on the original archive files and not have any hidden steps involving manually uncompressing files, etc.

Specific Dataset Issues
- bioasq_task_c_2017 issues per above
- PsyTAR assumes `data_dir` points directly to `PsyTAR_dataset.xlsx` (vs a directory containing that file).
- CAS and ESSAI assume `corpora.zip` is unzipped into `data_dir`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.