POSYDON-code / POSYDON-code/POSYDON
[BUG]: Issues with get-posydon-data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 45
- Forks
- 37
- Avg merge
- 8d 20h
- Merged PRs (30d)
- 5
Description
Describe the bug
Running get-posydon-data resulted in a partially downloaded file, as the connection to Zenodo servers hosting the datasets was interrupted. This leaves behind a partially downloaded .tar.gz dataset file. This forces the user to restart the get-posydon-data, but due to line 159-160 in posydon/utils/data_download.py, an error is raised for an existing .tar.gz file, causing the script to always raise an error and crash. The only way to fix it is to manually delete the .tar.gz file.
Furthermore, re-running get-posydon-data after an interrupted download causes the entire dataset to be redownloaded, instead of skipping the files that already exist.
The connection drop produces the following error:
File "/Users/anurag/.pyenv/versions/3.11.9/lib/python3.11/urllib/request.py", line 280, in urlretrieve raise ContentTooShortError( urllib.error.ContentTooShortError: <urlopen error retrieval incomplete: got only 6064525812 out of 10241444651 bytes>
And if the incomplete .tar.gz exists:
FileExistsError: POSYDON data already exists at /Users/anurag/Documents/POSYDON/posydon_data/POSYDON_data_DR2_grids_2Zsun.tar.gz.
To Reproduce
Run get-posydon-data, and interrupt the script (ex. ctrl+c) such that the data folder contains the unzipped .tar.gz file for a dataset. Running get-posydon-data reproduces this error.
Expected behavior
If the connection to Zenodo is dropped (raising the ContentTooShortError), instead of stopping the script completely, the script should re-try the download.
If any error is raised, such as the FileExistsError, it should delete the partially downloaded .tar.gz file or give the user the option to delete/rename/skip instead of stopping the script completely.
Some behavior should exist to determine if a dataset already exists, skipping it when re-running get-posydon-data, or giving the user the option to re-download it.
POSYDON Version:
V2.3.2
System Configuration:
MacOS
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the get-posydon-data entry point and lines 159-160 of posydon/utils/data_download.py, then reproduce the interrupted Zenodo download described in the issue. The work is done when retries handle ContentTooShortError, partial archives no longer block reruns, and already-present datasets can be skipped or re-downloaded without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100