Decouple original topic and compacted topic retention policies
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Is your feature request related to a problem? Please describe.**
When publishing to a topic that is compacted, the producer needs to publish the full state of the entity being published.
For example, it wouldn't make sense to publish to a compacted topic an event like "User added item to shopping cart" because a consumer wouldn't be able to reconstruct the state from only the last message for that user id. Each time the producer published, it would need to publish a message like "User shopping cart state".
Consuming from a compacted topic is not just a consumer decision. The producer also needs to allow for it in advance.
Pulsar has the following compacted topic support (https://pulsar.apache.org/docs/en/cookbooks-compaction/#when-should-i-use-compacted-topics):
> - They can read from the "original," non-compacted topic in case they need access to "historical" values, i.e. the entirety of the topic's messages.
> - They can read from the compacted topic if they only want to see the most up-to-date messages.
I intend to have all consumers of these messages on the compacted version of this topic. As such, it would be very wasteful to continue to store the complete history of messages on the original topic.
However, we can't set a retention policy on the original topic without that also affecting the compacted topic (https://pulsar.apache.org/docs/en/concepts-topic-compaction/#compaction):
> Topic compaction does, however, respect retention. If retention has removed a message from the message backlog of a topic, the message will also not be readable from the compacted topic ledger.
So it sounds like if I set a retention policy to remove old messages from the "original" topic, that will also remove messages from the compacted topic.
**Describe the solution you'd like**
I'd like storage to only be consumed for the compacted topic. Given nothing will be consuming from the original topic, it can be cleared.
**Describe alternatives you've considered**
While it is promoted as a feature, given the producers need to be aware the messages they are publishing will be compacted, I'm not sure what the benefit of having both the original and compacted topic. Could it be that the producers make a decision to only publish to a compacted topic?
**Additional context**
This was originally discussed on slack: https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1589871154137500
As for my use case, I'm currently evaluating Pulsar. Topic compaction and tiered storage are important and why I'm looking into pulsar.
Contributor guide
Research direction
Start with the linked Pulsar topic-compaction and retention documentation, then review the linked Slack discussion for prior context. Trace how retention currently affects both the original and compacted topic, and define how separate policies should behave for the stated use case. Done means the original topic can be cleared without removing readable data from the compacted topic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100