Graylog2 / Graylog2/graylog2-server

Support Throttling in Pulsar

Open
#9,786 2 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

The disk-based Kafka journal supports throttling of input transports. This issue serves as a starting point for a discussion about how, and *if* we would want to support this with a Pulsar based message queue.

### Current throttling behaviour with the disk-based Kafka journal

The current [implementation](https://github.com/Graylog2/graylog2-server/blob/f35df42e165ac570b8b27de3f8eeac85e74ed610/graylog2-server/src/main/java/org/graylog2/plugin/inputs/transports/ThrottleableTransport.java#L172) will throttle a transport based on a couple of criteria. They are almost exclusively based on observations of the journal state. We have listed the conditions in our [documentation](https://docs.graylog.org/en/4.0/pages/sending_data.html):
1. If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.
2. Throttling will occur if the Journal has more than 100k uncommitted entries.
3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).
4. Throttling will occur if the process ring buffer is full.
5. Nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.
6. Throttling will occur if the Journal is more than 90% full.
7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.

### Throttling with pulsar

We have to decide if it would make sense to replicate the current behaviour with pulsar.

> 1. If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.
> 2. Throttling will occur if the Journal has more than 100k uncommitted entries.
> 3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).

For conditions _1, 2 and 3_ we need to know how many uncommitted messages exist in a pulsar topic (the _backlog_). We can get this information with the admin client:
```
admin.topics().getStats("persistent://public/default/input-message-queue");
```
However, the `msgBacklog` metric we can get from the stats counts _batched_ messages. We cannot easily compute the number of individual messages from that, so we might want to use different numbers than `100k` and `20k` to base throttling decisions on.

> 4. Throttling will occur if the process ring buffer is full.

Condition _4_ is not affected by the journal implementation.

> 5. Nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.
> 6. [...]
> 7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.

For conditions _5 and 7_ we can measure read and write rates locally in our `PulsarMessageQueueWriter` and `PulsarMessageQueueReader`

> 6. Throttling will occur if the Journal is more than 90% full.

For condition _6_ we would need to specify a quota, in order to determine what "full" means. We can do this with e.g.
```
admin.namespaces().setBacklogQuota("public/default", new BacklogQuota(10 * 1024,
BacklogQuota.RetentionPolicy.producer_request_hold));
```
However, I don't see a straight-forward way to check the current status of the quota. Once the quota is exceeded, the producer will either be unable to send messages, or we have to configure pulsar to start deleting the oldest messages from the backlog.

Alternatively, we could look at `storageSize` in the topic stats and define a virtual limit that we'd want to use to trigger throttling. That would be quite similar to condition _2_ but using _bytes_ instead of _number of messages_ as the limit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.