Validate weather-dl target location before starting the job
Open
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
When the *weather-dl* job is submitted with an invalid path, e.g. a local path instead of GCS when using the Dataflow runner, the job never throws an error. Instead, it gets stuck in the exponential backoff loop of the `Fetcher.upload()` [method](https://github.com/google/weather-tools/blob/de4fa216e428666d183a31155862db6f4420e424/weather_dl/download_pipeline/fetcher.py#L57).
**Symptoms**
GCS write exponential backoff WARN message:
```
Retry with exponential backoff: waiting for 295.9412769663183 seconds before retrying open because we caught exception: FileNotFoundError: [Errno 2] No such file or directory: ''
```
Contributor guide
Assessment
This issue has not been assessed yet.