apache / apache/pulsar

Enhancement: Add message-count сompaction policy as alternatives to backlog-size threshold

Open
#26,091 1 comment 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

In topics where messages are roughly uniform in size, a byte-based threshold works as a reasonable approximation: bytes accumulated since last compaction correlates well enough with the number of new key updates.
This assumption breaks down when message sizes are heterogeneous. Consider a topic that mixes small control events (a few hundred bytes) with large snapshot messages (megabytes):

A low threshold is hit after just a handful of large snapshots, triggering compaction runs that are disproportionate to the actual number of stale keys — wasting broker I/O and ledger resources
A high threshold is never reached during stretches of small-message activity, even when thousands of key updates have accumulated and the compacted view is significantly out of date

The core mismatch is that bytes ≠ number of key updates. Compaction's purpose is to retain only the latest value per key; the amount of work it needs to do — and how stale the compacted view is — is determined by how many messages have been written since the last run, not by their total byte size.
A message-count threshold addresses this directly: compaction fires after N new messages regardless of their individual sizes, providing a stable and predictable trigger that reflects actual compaction pressure rather than storage volume.

### Solution

_No response_

### Alternatives

Tuning topicCompactionThresholdInBytes per topic — already possible, but operationally expensive on large clusters with heterogeneous topics and does not solve the fundamental mismatch between bytes and message staleness
External compaction triggering via Admin API — works as a workaround but requires additional infrastructure and loses the simplicity of built-in automation

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing topicCompactionThresholdInBytes configuration and compaction trigger implementation, then compare it with the external Admin API workaround described in the issue. Define how a message-count threshold should coexist with the byte threshold and identify the relevant tests; done means compaction can be triggered by the configured number of new messages without depending on message size.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.