apache / apache/beam

Kafka Streams runner: splittable DoFn

Open
#39,631 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The Kafka Streams runner does not support splittable `DoFn`, bounded or unbounded. Sources are read through the deprecated primitive `Read`, which the runner forces Read-based SDFs into during translation.

Supporting SDF means handling restrictions, checkpointing and residuals over the Fn API, along with dynamic splitting. The `UsesBoundedSplittableParDo` and `UsesUnboundedSplittableParDo` categories are excluded from the ValidatesRunner suite.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka Streams runner's translation of the deprecated Read primitive and the ValidatesRunner suite categories UsesBoundedSplittableParDo and UsesUnboundedSplittableParDo. Trace how the runner handles restrictions, checkpointing, residuals, and dynamic splitting through the Fn API. Done means both categories are supported rather than excluded, for bounded and unbounded splittable DoFns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.