Date ranges (e.g., 2014/2019) are incorrectly parsed as single year in metadata
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 559
- Forks
- 70
- Avg merge
- 1m
- Merged PRs (30d)
- 2
Description
Before opening an issue, please:
- Make sure you are using the latest version using
datasets --version
datasets version: 18.15.0 - Review our documentation
Describe the bug
Date ranges are not correctly downloaded with the datasets CLI. Sequences with collection date ranges in the format YYYY/YYYY (e.g., 2014/2019), which is a valid ENA format, are truncated to only the end year in the downloaded metadata.
For example, sequence MW179421.1 was collected between 2014 and 2019 (shown as 2014/2019 on GenBank), but datasets reports only year=2019 in the metadata file.
This affects all NCBI entries with this date format.
Impact: This is problematic for temporal analyses and phylodynamic inference, as the date uncertainty/range is lost, potentially biasing tip date calibrations and evolutionary rate estimates.
Indicate what operating system you're using
Linux (WSL)
To Reproduce
Steps to reproduce the behavior:
- Run:
datasets download virus genome taxon "39054" --no-progressbar --filename data/ncbi_dataset.zip - Unzip and open
data/ncbi_dataset/data/data_report.jsonl - Search for accession
MW179421.1and checkcollectionDate
Actual output:
"isolate": {
"collectionDate": "2019",
"name": "ZY2017-12-EV71"
}
Expected behavior
The date range should be preserved, either as:
- Original format:
"collectionDate": "2014/2019" - Structured fields:
"collectionDateStart": "2014","collectionDateEnd": "2019"
Currently, only the end year (2019) is retained, losing the 5-year uncertainty window.
Note: This issue appears specific to the / format for date ranges. Other range formats have not been tested.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the reported datasets download virus genome taxon command, then inspect data/ncbi_dataset/data/data_report.jsonl for accession MW179421.1. Compare collectionDate with the ENA value 2014/2019; the work is done when the downloaded metadata preserves the range, either in its original form or as explicit start and end fields.
Written by the indexing model from the issue text.
Assessment
- Domain
- cli, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100