google / google/weather-tools

Validate weather-dl target location before starting the job

Open
#209 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

When the *weather-dl* job is submitted with an invalid path, e.g. a local path instead of GCS when using the Dataflow runner, the job never throws an error. Instead, it gets stuck in the exponential backoff loop of the `Fetcher.upload()` [method](https://github.com/google/weather-tools/blob/de4fa216e428666d183a31155862db6f4420e424/weather_dl/download_pipeline/fetcher.py#L57).

**Symptoms**

GCS write exponential backoff WARN message:

```
Retry with exponential backoff: waiting for 295.9412769663183 seconds before retrying open because we caught exception: FileNotFoundError: [Errno 2] No such file or directory: ''
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.