bigscience-workshop / bigscience-workshop/biomedical
All local datasets need a list of filenames for downloading
- Dominant language
- Python
- Stars
- 505
- Forks
- 117
- PR merge metrics
- No merged PRs in 30d
Description
All local datasets need a list of the exact filenames to download. Several files don't have any indication of what a user needs to manually download. This requires slogging through the code to find out what is needed.
For example, consider [bioasq_task_c_2017](https://github.com/bigscience-workshop/biomedical/tree/master/bigbio/biodatasets/bioasq_task_c_2017) which requires some combination of
```
BioASQ_2017_Task5C_Final.tar.gz
BioASQ_2017_Task5C_Training.tar.gz
taskCTrainingData2017.json
taskc_golden2.json
```
and expects that the `tar.gz` files are all manually uncompressed before loading. None of these details are documented in the current script. Moreover, these are the types of requirements we should automate in the script directly.
I'd propose we require either:
- A list of the exact URLs to fetch (assuming the user is authenticated as needed)
- A list of the file names themselves
We should *always* operate on the original archive files and not have any hidden steps involving manually uncompressing files, etc.
Specific Dataset Issues
- bioasq_task_c_2017 issues per above
- PsyTAR assumes `data_dir` points directly to `PsyTAR_dataset.xlsx` (vs a directory containing that file).
- CAS and ESSAI assume `corpora.zip` is unzipped into `data_dir`
Contributor guide
Assessment
This issue has not been assessed yet.