Retry downloading a shard when file sizes are zero.
Open
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
Sometimes, files downloaded by weather-dl are created in GCS, but contain zero bytes. To address this, we could:
- Add a check after fetch is complete that checks the file size in the bucket. If zero, retry the DL.
- When we check if the file exists or needs to be skipped, we should also check the file size.
Workarounds: Users can make use of `gsutil` or equivalent tools to find all empty files. Then, they can delete these and re-download (with a separate invocation).
Contributor guide
Assessment
This issue has not been assessed yet.