Kafka Streams runner: read a source's splits in parallel
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
## Summary
The Kafka Streams runner splits a source into exactly one part and reads it with a single reader, so a source that could be read in parallel is not.
`ReadTranslator` calls `split(1, options)` and requires exactly one split back. A source returning more is rejected at translation rather than having the extra splits silently dropped, since that would be data loss, but the effect is that such a source cannot be read at all.
Reading several splits means giving each one its own reader and its own watermark, and distributing them across instances, which belongs with the topic-based shuffle work. Bounded and unbounded sources are both affected.
Part of #18479.
Contributor guide
Research direction
Start with ReadTranslator, where split(1, options) currently requires exactly one split, and read the topic-based shuffle work tracked in #18479. The change is done when bounded and unbounded sources can use multiple splits with separate readers and watermarks, distributed across Kafka Streams instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100