Graylog2 / Graylog2/graylog2-server

Pulsar queue naming

Open
#9,806 3 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

A Pulsar queue name consists of three parts: `tenant`, `namespace` and `topic`

For example: `persistent://public/default/my-topic`

## Tenants
- The "users" within Pulsar.
- We want to create one tenant for each customer, therefore using the graylog `cluster-id` is an obvious candidate.
- We will likely additionally authenticate each tenant (from the pulsar client). As an additional safe-guard.

## Namespaces
- Are needed to group topics
- Most settings have to be applied on a namespace level, (not per topic)
- You can assign each user permissions to certain `namespaces` (not topics)
- We currently have only one use case for Pulsar, that's why I propose using only one namespace named `inputs`
reflecting our use for message inputs

## Topics

- Topics are the actual "target" you produce / subscribe messages to/from.
- Topics can be auto-generated from the producer/consumer, but that will only create simple topics, not `partitioned topics` which we will probably need to scale out. Therefore, we need to pre-create partitioned topics when we are rolling out a new customer via the Pulsar admin API.
- Addition: This can be controlled via the `broker.conf`:
```# Enable topic auto creation if new producer or consumer connected (disable auto creation with value false)
allowAutoTopicCreation=true

# The type of topic that is allowed to be automatically created.(partitioned/non-partitioned)
allowAutoTopicCreationType=non-partitioned

# Enable subscription auto creation if new consumer connected (disable auto creation with value false)
allowAutoSubscriptionCreation=true

# The number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned.
defaultNumPartitions=1
```
- Pulsar consumers can subscribe to multiple topics. -> http://pulsar.apache.org/docs/en/client-libraries-java/#multi-topic-subscriptions
This would give us the possibility to produce (write) into multiple topics, but read from all of them with a single consumer.
E.g. we could choose build topics by combining the graylog `node-id` and the `input-id` to `-`
and then subscribe to `-.*`
However, since we currently only want not to mix the messages over multiple processing nodes, it is sufficient to just
use one topic for each graylog node. Thus the `topic` name would become the graylog `node-id`

## Proposal
```
"tenant" "namespace" "topic"
/ inputs /
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.