When streaming jobs with `weather-mv`, should not require URIs exist before launching the streaming
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
When the user launches a `weather-mv` job with topics (meaning it's watching for streaming events like new objects being created), it is possible that the provided URI pattern doesn't match any existing objects in the bucket yet.
However, we currently always [require all URIs exist](https://github.com/google/weather-tools/blob/main/weather_mv/loader_pipeline/pipeline.py#L51) before we launch the beam job [code](https://github.com/google/weather-tools/blob/main/weather_mv/loader_pipeline/pipeline.py#L64-L66).
This will cause immediate failures for the user so they are not able to launch the streaming job at all.
The check for all URIs exist should only happen for the case when no topics are created. Similarly, for the `first_uri` argument, we only use it for the non-streaming case. With `topics`, we should not create `first_uri` to be clearer.
Contributor guide
Assessment
This issue has not been assessed yet.