apache / apache/beam

Kafka Streams runner: stateful ParDo and user timers

Open
#39,629 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The Kafka Streams runner does not support user state or user timers in `ParDo`. Stages run with `StateRequestHandler.unsupported()` and no timer receivers, and the corresponding `@ValidatesRunner` categories are excluded.

This covers `UsesStatefulParDo`, `UsesTimersInParDo`, `UsesTimerMap`, `UsesLoopingTimer`, `UsesProcessingTimeTimers`, `UsesOnWindowExpiration` and the various state types.

The pieces to build on already exist: `KafkaStreamsStateInternals` and `KafkaStreamsTimerInternals` back the windowed GroupByKey over Kafka Streams state stores, and the same backing could serve user state once the harness side is wired up. Input would also have to be shuffled by key, as GroupByKey already does.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with KafkaStreamsStateInternals and KafkaStreamsTimerInternals, then trace the Kafka Streams runner stages using StateRequestHandler.unsupported() and no timer receivers. Review the excluded @ValidatesRunner categories and existing windowed GroupByKey state-store behavior. Done means user state and timers in ParDo are supported, including keyed input handling, with the listed validation categories enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.