KafkaIO should support dynamic addition of Kafka partitions to assigned topics.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Kafka topics may add partitions dynamically (doesn't require Kafka to restart, or halt a topic), and the KafkaIO should probably support this.
*Note:*
Consistently assigning partitions should be taken into account, specifically for the case of reading from multiple topics, where one (or more) of the topics added partitions while the pipeline is running.
Imported from Jira [BEAM-727](https://issues.apache.org/jira/browse/BEAM-727). Original Jira may contain additional context.
Reported by: amitsela.
Contributor guide
Research direction
Start by locating the KafkaIO entry point and its partition-assignment logic. Investigate how assigned topics are monitored while a pipeline runs, including multiple topics, and define done as consistently assigning newly added partitions without restarting Kafka or halting the topic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100