apache / apache/beam

Kafka Streams runner: expose topic-level configuration for runner-created topics

Open
#39,565 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

## Summary

The Kafka Streams runner creates its own internal topics (a bootstrap topic per Impulse and Read, a repartition topic per GroupByKey). Today only the partition count (`--internalParallelism`) and replication factor are configurable.

A user should be able to set the rest of the topic-level configuration — retention, compaction, segment settings, min.insync.replicas and so on. Two options discussed on https://github.com/apache/beam/pull/39546: pass a JSON-serialized dictionary of topic configs, and/or expose the most common ones as separate pipeline options.

Part of #18479.

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka Streams runner and its handling of runner-created bootstrap and repartition topics for Impulse, Read, and GroupByKey. Review the existing --internalParallelism and replication-factor pipeline options, then compare the JSON topic-config and separate-option approaches discussed in PR 39546. Done means supported topic-level settings are exposed and applied to those internal topics, with tests covering the configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.