Graylog2 / Graylog2/graylog2-server
Pulsar queue naming
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
A Pulsar queue name consists of three parts: `tenant`, `namespace` and `topic`
For example: `persistent://public/default/my-topic`
## Tenants
- The "users" within Pulsar.
- We want to create one tenant for each customer, therefore using the graylog `cluster-id` is an obvious candidate.
- We will likely additionally authenticate each tenant (from the pulsar client). As an additional safe-guard.
## Namespaces
- Are needed to group topics
- Most settings have to be applied on a namespace level, (not per topic)
- You can assign each user permissions to certain `namespaces` (not topics)
- We currently have only one use case for Pulsar, that's why I propose using only one namespace named `inputs`
reflecting our use for message inputs
## Topics
- Topics are the actual "target" you produce / subscribe messages to/from.
- Topics can be auto-generated from the producer/consumer, but that will only create simple topics, not `partitioned topics` which we will probably need to scale out. Therefore, we need to pre-create partitioned topics when we are rolling out a new customer via the Pulsar admin API.
- Addition: This can be controlled via the `broker.conf`:
```# Enable topic auto creation if new producer or consumer connected (disable auto creation with value false)
allowAutoTopicCreation=true
# The type of topic that is allowed to be automatically created.(partitioned/non-partitioned)
allowAutoTopicCreationType=non-partitioned
# Enable subscription auto creation if new consumer connected (disable auto creation with value false)
allowAutoSubscriptionCreation=true
# The number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned.
defaultNumPartitions=1
```
- Pulsar consumers can subscribe to multiple topics. -> http://pulsar.apache.org/docs/en/client-libraries-java/#multi-topic-subscriptions
This would give us the possibility to produce (write) into multiple topics, but read from all of them with a single consumer.
E.g. we could choose build topics by combining the graylog `node-id` and the `input-id` to `-`
and then subscribe to `-.*`
However, since we currently only want not to mix the messages over multiple processing nodes, it is sufficient to just
use one topic for each graylog node. Thus the `topic` name would become the graylog `node-id`
## Proposal
```
"tenant" "namespace" "topic"
/ inputs /
```
Contributor guide
Assessment
This issue has not been assessed yet.