google / google/weather-tools

Retry downloading a shard when file sizes are zero.

Open
#261 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

Sometimes, files downloaded by weather-dl are created in GCS, but contain zero bytes. To address this, we could:
- Add a check after fetch is complete that checks the file size in the bucket. If zero, retry the DL.
- When we check if the file exists or needs to be skipped, we should also check the file size.

Workarounds: Users can make use of `gsutil` or equivalent tools to find all empty files. Then, they can delete these and re-download (with a separate invocation).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.