apache / apache/beam

Support AfterPane.elementGapAtMost() trigger and its combination with elementCountAtLeast()

Open
#18,329 0 comments 0 reactions 0 assignees View on GitHub
core java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

We need a timestamp-driven trigger to use as a cheaper (or more efficient) version of the ProcessingTime trigger.

The problem of using ProcessingTime trigger is that current runners' supports are not very efficient, and couldn't work for pipelines that have lots of keys (for example, flink runner will scan timers for all keys when watermark advance).

We have used AfterPane.elementGapAtMost() trigger in our production, and want to merge it back. And, we believe it could be the solution for people who have the similar issue.

Implementation for reference:
https://github.com/apache/beam/compare/master...peihe:custom-after-pane?expand=1

Imported from Jira [BEAM-2402](https://issues.apache.org/jira/browse/BEAM-2402). Original Jira may contain additional context.
Reported by: peihe0@gmail.com.

Contributor guide

Open the contributing guide

Research direction

Start with the linked custom-after-pane implementation and the AfterPane.elementGapAtMost() and elementCountAtLeast() entry points. Compare that reference with Beam's existing ProcessingTime trigger behavior, then determine the tests needed for the trigger and their combination. Done means the requested timestamp-driven trigger is supported without the stated per-key timer-scan cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.