Support multi-topic subscriptions with Apache Pulsar
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
There are multiple ways how this could be implemented, namely:
1. Instead of/Beyond the existing configuration option `"stream.pulsar.topic.name": "pulsar-test"` we could have something like `"stream.pulsar.topic.names": [ "pulsar-test-1", "pulsar-test-2"]`.
2. Beyond the above existing configuration option we could have something like `"stream.pulsar.topic.pattern": "pulsar-test-*"` where the pattern adheres to standard regex rules.
3. We could support such pattern-like inputs in the existing configuration option, e.g. an input field like `"stream.pulsar.topic.name": "persistent://some_tenant/some_namespace/*"` should make Pinot bind a real-time table to all topics under the given tenant/namespace.
Another alternative solution would be the capability of binding multiple topics to the same real-time table by sending subsequent requests to the Pinot Server (e.g., first, creating `some_table` as a real-time one and providing topic `A` as its source, however, a subsequent real-time table request with the name `some_table` and topic `B` would just subscribe to `B` and direct the corresponding flow into `some_table`, as well).
Contributor guide
Research direction
Start by comparing the proposed configuration forms for multiple Apache Pulsar topics: a topic-name list, a topic pattern, wildcard input, and subsequent requests against one real-time table. No files, tests, or implementation entry points are identified in the issue, so first locate the existing stream.pulsar.topic.name handling and its real-time table request path. Done should mean one agreed approach is implemented and verified for multiple topic subscriptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100