apache / apache/beam

Kafka Streams runner: side inputs

Open
#39,628 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The Kafka Streams runner executes stages with `StateRequestHandler.unsupported()`, so a pipeline using side inputs cannot run.

Supporting them means serving side input state to the SDK harness while a bundle is in flight, and holding the stage's watermark until each side input is ready. `BatchSideInputHandlerFactory` and `StateRequestHandlers.forSideInputHandlerFactory` are how the Flink runner does it.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka Streams runner code that currently uses StateRequestHandler.unsupported(), then read BatchSideInputHandlerFactory and StateRequestHandlers.forSideInputHandlerFactory as the Flink reference. The work is done when side input state is served to the SDK harness during an active bundle and the stage watermark waits for each side input to be ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.