[Enhancement] Optimize delayed message delivery by reducing unnecessary reads to storage in InMemoryDelayedDeliveryTracker
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
In the current delayed message delivery, there's an opportunity to reduce unnecessary reads to storage.
In Pulsar, there are 2 implementations for the delayed delivery tracker, InMemoryDelayedDeliveryTracker and BucketDelayedDeliveryTracker.
This is configured by the `delayedDeliveryTrackerFactoryClassName` configuration key. The default setting chooses the in memory implementation:
https://github.com/apache/pulsar/blob/b02d52ca8c8afd62681c0e243d16d8958abb5380/conf/broker.conf#L614-L617
The BucketDelayedDeliveryTracker contains an optimization to skip messages in reading which have been "indexed":
https://github.com/apache/pulsar/blob/11a615e7cecb57c8c240cc565c5ac62d938c490d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L446-L457
This already reduces reads when BucketDelayedDeliveryTracker is in use.
The state of the InMemoryDelayedDeliveryTracker gets cleared after all consumers have disconnected:
https://github.com/apache/pulsar/blob/12b057999f5674ae6da6b04ff8bd212d1904cc88/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L187-L202
### Solution
It would be useful to keep state also in the InMemoryDelayedDeliveryTracker and skip reading delayed messages when the information is already available for the delivery time of a specific entry.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with PersistentDispatcherMultipleConsumers.java, especially the delayed-message read optimization around lines 446-457 and the state-clearing logic around lines 187-202. Read the InMemoryDelayedDeliveryTracker and BucketDelayedDeliveryTracker implementations, then verify that known delivery-time information lets the in-memory tracker skip unnecessary storage reads. Done means the optimization applies to the in-memory implementation without changing the configured tracker behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100