Stop streaming pipeline when global watermark reaches +INF
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Stop streaming pipeline when global watermark reaches ****INF as suggested in BEAM-849 comments.
We can do this similarly to the solution in [TestSparkRunner#awaitWatermarksOrTimeout](https://github.com/apache/beam/blob/v2.1.0/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java#L158-L179) by polling ```
SparkTimerInternals.global(GlobalWatermarkHolder.get(...))
```
We can also remove this code from `TestSparkRunner` if we make sure all sources in test behave in a way which shifts the watermark to ****INF at the end of the test.
Imported from Jira [BEAM-2847](https://issues.apache.org/jira/browse/BEAM-2847). Original Jira may contain additional context.
Reported by: aviemzur.
Contributor guide
Research direction
Start with BEAM-849 for the intended watermark behavior, then read TestSparkRunner#awaitWatermarksOrTimeout and the referenced SparkTimerInternals.global(GlobalWatermarkHolder.get(...)) path. Confirm how the streaming pipeline observes a global watermark of +INF and define completion by ensuring the pipeline stops then; remove the TestSparkRunner logic only if the source behavior is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100