apache / apache/beam

[Kafka Streams runner] Use a single Impulse bootstrap topic with fan-out

Open
#39,382 1 comment 0 reactions 1 assignee Claimed by @junaiddshaukat View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

## Summary

Currently every Impulse transform gets its own bootstrap topic (`__beam_impulse__`), because Kafka Streams rejects registering one topic on two source nodes and a pipeline can contain several Impulses (e.g. an empty `Create` plus the dummy branch `PAssert` adds).

Per review of #39380 this can be optimized: keep a single per-application bootstrap topic with one source node, and fan the created element out to all impulse processors from there, instead of one topic per transform.

Part of #18479.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.