apache / apache/beam

Kafka Streams runner: lifecycle management for runner-created topics

Open
#39,566 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

## Summary

The Kafka Streams runner creates the internal topics a pipeline needs implicitly, when the job starts (https://github.com/apache/beam/pull/39546).

It may be better to make that explicit and give the topics a lifecycle, so an operator can create them ahead of time and clean them up afterwards:

```
# just create topics
java -cp ... my.class --runner=KafkaStreams [... other opts ...] --create-topics
# run
java -cp ... my.class --runner=KafkaStreams [... other opts ...]
# delete runner-created topics
java -cp ... my.class --runner=KafkaStreams [... other opts ...] --delete-topics
```

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start by reading PR #39546 for the current Kafka Streams runner topic behavior and issue #18479 for the broader context. Define the lifecycle requirements for the --create-topics, normal run, and --delete-topics commands, with completion demonstrated by operators being able to create and remove runner-created topics explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.