ncbi / ncbi/datasets

Date ranges (e.g., 2014/2019) are incorrectly parsed as single year in metadata

Open
#557 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Jupyter Notebook
Stars
559
Forks
70
Avg merge
1m
Merged PRs (30d)
2

Description

Before opening an issue, please:
  • Make sure you are using the latest version using datasets --version
    datasets version: 18.15.0
  • Review our documentation

Describe the bug

Date ranges are not correctly downloaded with the datasets CLI. Sequences with collection date ranges in the format YYYY/YYYY (e.g., 2014/2019), which is a valid ENA format, are truncated to only the end year in the downloaded metadata.

For example, sequence MW179421.1 was collected between 2014 and 2019 (shown as 2014/2019 on GenBank), but datasets reports only year=2019 in the metadata file.

This affects all NCBI entries with this date format.

Impact: This is problematic for temporal analyses and phylodynamic inference, as the date uncertainty/range is lost, potentially biasing tip date calibrations and evolutionary rate estimates.

Indicate what operating system you're using

Linux (WSL)

To Reproduce

Steps to reproduce the behavior:

  1. Run:
    datasets download virus genome taxon "39054" --no-progressbar --filename data/ncbi_dataset.zip
    
  2. Unzip and open data/ncbi_dataset/data/data_report.jsonl
  3. Search for accession MW179421.1 and check collectionDate

Actual output:

"isolate": {
  "collectionDate": "2019",
  "name": "ZY2017-12-EV71"
}

Expected behavior

The date range should be preserved, either as:

  • Original format: "collectionDate": "2014/2019"
  • Structured fields: "collectionDateStart": "2014", "collectionDateEnd": "2019"

Currently, only the end year (2019) is retained, losing the 5-year uncertainty window.

Note: This issue appears specific to the / format for date ranges. Other range formats have not been tested.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the reported datasets download virus genome taxon command, then inspect data/ncbi_dataset/data/data_report.jsonl for accession MW179421.1. Compare collectionDate with the ENA value 2014/2019; the work is done when the downloaded metadata preserves the range, either in its original form or as explicit start and end fields.

Written by the indexing model from the issue text.

Assessment

Domain
cli, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.