apache / apache/beam

[Feature Request]: Expose TimerStateInternals.currentOutputWatermarkTime to allow for DoFns to handle elements behind the watemark differently

Open
#23,379 16 comments 0 reactions 0 assignees View on GitHub
core java new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Currently element lateness is evaluated during windowing/triggering based upon TimerInternals.currentOutputWatermarkTime

There are some cases where users want to handle late records differently. For example, instead of dropping late events or assigning them to their original (now late window) with allowedLateness, the late record could be assigned to a different window, logged and monitored explicitly, or mutated and output differently.

By exposing the currentOutputWatermarkTime on the ProcessContext it would be possible for a DoFn to compare the timestamp of the incoming watermark to it to perform custom handling for late events.
The benefit of exposing the timestamp and not just an isLate method is that it would also be possible for a DoFn to set the timestamp of their output to a guaranteed non-late time based upon this value.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-java-core

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.