apache / apache/pulsar

[Enhancement] Optimize delayed message delivery by reducing unnecessary reads to storage in InMemoryDelayedDeliveryTracker

Open
#23,912 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

In the current delayed message delivery, there's an opportunity to reduce unnecessary reads to storage.

In Pulsar, there are 2 implementations for the delayed delivery tracker, InMemoryDelayedDeliveryTracker and BucketDelayedDeliveryTracker.
This is configured by the `delayedDeliveryTrackerFactoryClassName` configuration key. The default setting chooses the in memory implementation:
https://github.com/apache/pulsar/blob/b02d52ca8c8afd62681c0e243d16d8958abb5380/conf/broker.conf#L614-L617

The BucketDelayedDeliveryTracker contains an optimization to skip messages in reading which have been "indexed":
https://github.com/apache/pulsar/blob/11a615e7cecb57c8c240cc565c5ac62d938c490d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L446-L457

This already reduces reads when BucketDelayedDeliveryTracker is in use.

The state of the InMemoryDelayedDeliveryTracker gets cleared after all consumers have disconnected:
https://github.com/apache/pulsar/blob/12b057999f5674ae6da6b04ff8bd212d1904cc88/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L187-L202

### Solution

It would be useful to keep state also in the InMemoryDelayedDeliveryTracker and skip reading delayed messages when the information is already available for the delivery time of a specific entry.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with PersistentDispatcherMultipleConsumers.java, especially the delayed-message read optimization around lines 446-457 and the state-clearing logic around lines 187-202. Read the InMemoryDelayedDeliveryTracker and BucketDelayedDeliveryTracker implementations, then verify that known delivery-time information lets the in-memory tracker skip unnecessary storage reads. Done means the optimization applies to the in-memory implementation without changing the configured tracker behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.