google / google/weather-tools

Crash weather-dl pipeline early for common class of errors.

Open
#223 0 comments 0 reactions 0 assignees View on GitHub
help wanted weather-dl
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

Let's distinguish between at least two types of errors in the `weather-dl` pipeline: The first are pipeline execution errors. For example, the network could go down, or a shard of data could be corrupted during transfer. For these, we should retry a request with our current fetching logic. The second type of errors are user-produced errors. Issues like not setting up enough disk space, typing in a variable incorrectly s.t. EC doesn't know what data to give, etc. — these should fail the pipeline as soon as possible. Thus, we need to devise a way to not retry these types of errors and fail the pipeline, so we get an early feedback signal to fix it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.