Kafka Streams runner: side inputs
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The Kafka Streams runner executes stages with `StateRequestHandler.unsupported()`, so a pipeline using side inputs cannot run.
Supporting them means serving side input state to the SDK harness while a bundle is in flight, and holding the stage's watermark until each side input is ready. `BatchSideInputHandlerFactory` and `StateRequestHandlers.forSideInputHandlerFactory` are how the Flink runner does it.
Part of #18479.
Contributor guide
Research direction
Start with the Kafka Streams runner code that currently uses StateRequestHandler.unsupported(), then read BatchSideInputHandlerFactory and StateRequestHandlers.forSideInputHandlerFactory as the Flink reference. The work is done when side input state is served to the SDK harness during an active bundle and the stage watermark waits for each side input to be ready.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100