Crash weather-dl pipeline early for common class of errors.
Open
help wanted
weather-dl
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
Let's distinguish between at least two types of errors in the `weather-dl` pipeline: The first are pipeline execution errors. For example, the network could go down, or a shard of data could be corrupted during transfer. For these, we should retry a request with our current fetching logic. The second type of errors are user-produced errors. Issues like not setting up enough disk space, typing in a variable incorrectly s.t. EC doesn't know what data to give, etc. — these should fail the pipeline as soon as possible. Thus, we need to devise a way to not retry these types of errors and fail the pipeline, so we get an early feedback signal to fix it.
Contributor guide
Assessment
This issue has not been assessed yet.