Kafka Streams runner: splittable DoFn
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The Kafka Streams runner does not support splittable `DoFn`, bounded or unbounded. Sources are read through the deprecated primitive `Read`, which the runner forces Read-based SDFs into during translation.
Supporting SDF means handling restrictions, checkpointing and residuals over the Fn API, along with dynamic splitting. The `UsesBoundedSplittableParDo` and `UsesUnboundedSplittableParDo` categories are excluded from the ValidatesRunner suite.
Part of #18479.
Contributor guide
Research direction
Start with the Kafka Streams runner's translation of the deprecated Read primitive and the ValidatesRunner suite categories UsesBoundedSplittableParDo and UsesUnboundedSplittableParDo. Trace how the runner handles restrictions, checkpointing, residuals, and dynamic splitting through the Fn API. Done means both categories are supported rather than excluded, for bounded and unbounded splittable DoFns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100