Kafka Streams runner: stateful ParDo and user timers
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The Kafka Streams runner does not support user state or user timers in `ParDo`. Stages run with `StateRequestHandler.unsupported()` and no timer receivers, and the corresponding `@ValidatesRunner` categories are excluded.
This covers `UsesStatefulParDo`, `UsesTimersInParDo`, `UsesTimerMap`, `UsesLoopingTimer`, `UsesProcessingTimeTimers`, `UsesOnWindowExpiration` and the various state types.
The pieces to build on already exist: `KafkaStreamsStateInternals` and `KafkaStreamsTimerInternals` back the windowed GroupByKey over Kafka Streams state stores, and the same backing could serve user state once the harness side is wired up. Input would also have to be shuffled by key, as GroupByKey already does.
Part of #18479.
Contributor guide
Research direction
Start with KafkaStreamsStateInternals and KafkaStreamsTimerInternals, then trace the Kafka Streams runner stages using StateRequestHandler.unsupported() and no timer receivers. Review the excluded @ValidatesRunner categories and existing windowed GroupByKey state-store behavior. Done means user state and timers in ParDo are supported, including keyed input handling, with the listed validation categories enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100