Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
In topics where messages are roughly uniform in size, a byte-based threshold works as a reasonable approximation: bytes accumulated since last compaction correlates well enough with the number of new key updates.
This assumption breaks down when message sizes are heterogeneous. Consider a topic that mixes small control events (a few hundred bytes) with large snapshot messages (megabytes):
A low threshold is hit after just a handful of large snapshots, triggering compaction runs that are disproportionate to the actual number of stale keys — wasting broker I/O and ledger resources
A high threshold is never reached during stretches of small-message activity, even when thousands of key updates have accumulated and the compacted view is significantly out of date
The core mismatch is that bytes ≠ number of key updates. Compaction's purpose is to retain only the latest value per key; the amount of work it needs to do — and how stale the compacted view is — is determined by how many messages have been written since the last run, not by their total byte size.
A message-count threshold addresses this directly: compaction fires after N new messages regardless of their individual sizes, providing a stable and predictable trigger that reflects actual compaction pressure rather than storage volume.
### Solution
_No response_
### Alternatives
Tuning topicCompactionThresholdInBytes per topic — already possible, but operationally expensive on large clusters with heterogeneous topics and does not solve the fundamental mismatch between bytes and message staleness
External compaction triggering via Admin API — works as a workaround but requires additional infrastructure and loses the simplicity of built-in automation
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reading the existing topicCompactionThresholdInBytes configuration and compaction trigger implementation, then compare it with the external Admin API workaround described in the issue. Define how a message-count threshold should coexist with the byte threshold and identify the relevant tests; done means compaction can be triggered by the configured number of new messages without depending on message size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100