Kafka Streams runner: lifecycle management for runner-created topics
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
## Summary
The Kafka Streams runner creates the internal topics a pipeline needs implicitly, when the job starts (https://github.com/apache/beam/pull/39546).
It may be better to make that explicit and give the topics a lifecycle, so an operator can create them ahead of time and clean them up afterwards:
```
# just create topics
java -cp ... my.class --runner=KafkaStreams [... other opts ...] --create-topics
# run
java -cp ... my.class --runner=KafkaStreams [... other opts ...]
# delete runner-created topics
java -cp ... my.class --runner=KafkaStreams [... other opts ...] --delete-topics
```
Part of #18479.
Contributor guide
Research direction
Start by reading PR #39546 for the current Kafka Streams runner topic behavior and issue #18479 for the broader context. Define the lifecycle requirements for the --create-topics, normal run, and --delete-topics commands, with completion demonstrated by operators being able to create and remove runner-created topics explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100