apache / apache/beam

Kafka Streams runner: read a source's splits in parallel

Open
#39,626 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

## Summary

The Kafka Streams runner splits a source into exactly one part and reads it with a single reader, so a source that could be read in parallel is not.

`ReadTranslator` calls `split(1, options)` and requires exactly one split back. A source returning more is rejected at translation rather than having the extra splits silently dropped, since that would be data loss, but the effect is that such a source cannot be read at all.

Reading several splits means giving each one its own reader and its own watermark, and distributing them across instances, which belongs with the topic-based shuffle work. Bounded and unbounded sources are both affected.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with ReadTranslator, where split(1, options) currently requires exactly one split, and read the topic-based shuffle work tracked in #18479. The change is done when bounded and unbounded sources can use multiple splits with separate readers and watermarks, distributed across Kafka Streams instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data, distributed-systems, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.