google / google/weather-tools

When streaming jobs with `weather-mv`, should not require URIs exist before launching the streaming

Open
#311 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

When the user launches a `weather-mv` job with topics (meaning it's watching for streaming events like new objects being created), it is possible that the provided URI pattern doesn't match any existing objects in the bucket yet.

However, we currently always [require all URIs exist](https://github.com/google/weather-tools/blob/main/weather_mv/loader_pipeline/pipeline.py#L51) before we launch the beam job [code](https://github.com/google/weather-tools/blob/main/weather_mv/loader_pipeline/pipeline.py#L64-L66).

This will cause immediate failures for the user so they are not able to launch the streaming job at all.

The check for all URIs exist should only happen for the case when no topics are created. Similarly, for the `first_uri` argument, we only use it for the non-streaming case. With `topics`, we should not create `first_uri` to be clearer.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.