apache / apache/beam

Merging of watermark holds depends on runner-specific "take the minimum" behavior

Open
#17,967 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When multiple watermark holds are to be combined via an OutputTimeFn and committed to persistent storage, the case where the OutputTimeFn computes the minimum is elided, assuming the underlying runner will perform this combine by default.

This is a natural default, but today it is a runner-specific behavior that the runner-agnostic code relies upon.

Imported from Jira [BEAM-24](https://issues.apache.org/jira/browse/BEAM-24). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start by locating the runner-agnostic code that combines watermark holds through an OutputTimeFn and persists the result. Compare the minimum case with other combine behaviors and identify where runner-specific behavior is assumed. Done means the minimum case is handled without relying on a runner-specific default, with relevant tests covering the merge.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.