apache / apache/pinot

Support multi-topic subscriptions with Apache Pulsar

Open
#8,806 3 comments 1 reaction 0 assignees View on GitHub
ingestion
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

There are multiple ways how this could be implemented, namely:

1. Instead of/Beyond the existing configuration option `"stream.pulsar.topic.name": "pulsar-test"` we could have something like `"stream.pulsar.topic.names": [ "pulsar-test-1", "pulsar-test-2"]`.
2. Beyond the above existing configuration option we could have something like `"stream.pulsar.topic.pattern": "pulsar-test-*"` where the pattern adheres to standard regex rules.
3. We could support such pattern-like inputs in the existing configuration option, e.g. an input field like `"stream.pulsar.topic.name": "persistent://some_tenant/some_namespace/*"` should make Pinot bind a real-time table to all topics under the given tenant/namespace.

Another alternative solution would be the capability of binding multiple topics to the same real-time table by sending subsequent requests to the Pinot Server (e.g., first, creating `some_table` as a real-time one and providing topic `A` as its source, however, a subsequent real-time table request with the name `some_table` and topic `B` would just subscribe to `B` and direct the corresponding flow into `some_table`, as well).

Contributor guide

Open the contributing guide

Research direction

Start by comparing the proposed configuration forms for multiple Apache Pulsar topics: a topic-name list, a topic pattern, wildcard input, and subsequent requests against one real-time table. No files, tests, or implementation entry points are identified in the issue, so first locate the existing stream.pulsar.topic.name handling and its real-time table request path. Done should mean one agreed approach is implemented and verified for multiple topic subscriptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.