apache / apache/beam

Kafka Streams runner: merging windows and custom WindowFns

Open
#39,630 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Two related windowing gaps in the Kafka Streams runner.

**Merging windows.** Session windows do not work. `ReduceFnRunner` drives merging through a merging window set that moves per-window state as windows merge, which the runner's state backing does not implement. `GroupByKeyTest$WindowTests.testGroupByKeyMergingWindows` and `CombineTest$WindowingTests.testSessionsCombine` are sickbayed for this.

**Custom WindowFns.** The runner rebuilds the standard WindowFns from their URNs and interprets them directly, which is what makes a pipeline from any SDK work. A WindowFn the user wrote is opaque to the runner and would have to be executed through the SDK harness that owns it.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with ReduceFnRunner, the Kafka Streams runner's state backing, and the merging-window behavior described in GroupByKeyTest$WindowTests.testGroupByKeyMergingWindows and CombineTest$WindowingTests.testSessionsCombine. Reproduce the sickbayed session-window tests first; done requires merging windows to work and a defined path for custom WindowFns through the owning SDK harness.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.