POSYDON-code / POSYDON-code/POSYDON

[BUG]: Issues with get-posydon-data

Open
#930 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
45
Forks
37
Avg merge
8d 20h
Merged PRs (30d)
5

Description

Describe the bug
Running get-posydon-data resulted in a partially downloaded file, as the connection to Zenodo servers hosting the datasets was interrupted. This leaves behind a partially downloaded .tar.gz dataset file. This forces the user to restart the get-posydon-data, but due to line 159-160 in posydon/utils/data_download.py, an error is raised for an existing .tar.gz file, causing the script to always raise an error and crash. The only way to fix it is to manually delete the .tar.gz file.

Furthermore, re-running get-posydon-data after an interrupted download causes the entire dataset to be redownloaded, instead of skipping the files that already exist.

The connection drop produces the following error:

File "/Users/anurag/.pyenv/versions/3.11.9/lib/python3.11/urllib/request.py", line 280, in urlretrieve raise ContentTooShortError( urllib.error.ContentTooShortError: <urlopen error retrieval incomplete: got only 6064525812 out of 10241444651 bytes>

And if the incomplete .tar.gz exists:

FileExistsError: POSYDON data already exists at /Users/anurag/Documents/POSYDON/posydon_data/POSYDON_data_DR2_grids_2Zsun.tar.gz.

To Reproduce
Run get-posydon-data, and interrupt the script (ex. ctrl+c) such that the data folder contains the unzipped .tar.gz file for a dataset. Running get-posydon-data reproduces this error.

Expected behavior
If the connection to Zenodo is dropped (raising the ContentTooShortError), instead of stopping the script completely, the script should re-try the download.

If any error is raised, such as the FileExistsError, it should delete the partially downloaded .tar.gz file or give the user the option to delete/rename/skip instead of stopping the script completely.

Some behavior should exist to determine if a dataset already exists, skipping it when re-running get-posydon-data, or giving the user the option to re-download it.

POSYDON Version:
V2.3.2

System Configuration:
MacOS

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the get-posydon-data entry point and lines 159-160 of posydon/utils/data_download.py, then reproduce the interrupted Zenodo download described in the issue. The work is done when retries handle ContentTooShortError, partial archives no longer block reruns, and already-present datasets can be skipped or re-downloaded without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.