[improve][broker] Speed up PendingAcksMap ack tracking with primitive storage and 50%+ lower memory
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the issues and found nothing similar.
### Motivation
`PendingAcksMap` is maintained per consumer and is on the broker-side consumer ack path. It tracks pending acknowledgements by `(ledgerId, entryId)` and is touched by dispatch, individual ack, partial ack, redelivery, and mark-delete cleanup.
The current implementation uses nested ordered maps with boxed keys and object values. This keeps the code simple, but it also adds per-entry object overhead, allocation rate, and GC pressure when a consumer has a large pending-ack window.
This issue tracks a series of focused improvements to reduce that overhead without changing consumer ack semantics.
A full draft implementation is available for review and validation:
https://github.com/apache/pulsar/pull/26024
### Solution
The main insight is that `PendingAcksMap` needs ordering at the ledger level, but not for every entry within a ledger.
The outer ledger map must stay ordered because `removeAllUpTo(markDeletePosition)` removes whole ledger buckets by ledger range. Inside one ledger, the common operations are exact lookup, exact remove, remaining-unacked update, and boundary cleanup for `entryId <= markDeleteEntryId`. These operations do not require the inner map to iterate entries in sorted order.
Redelivery ordering is not inherited from the per-ledger pending-ack map. The redelivery path rebuilds ordered positions separately, so replacing the inner ordered map does not change the redelivery ordering contract.
The split is:
| Part | PR | Scope |
| --- | --- | --- |
| 1 | #26028 | Add local primitive `Long2LongOpenHashMap` |
| 2 | #26030 | Pack `PendingAcksMap` values into `long` and use primitive accessors in `Consumer` |
| 3 | Follow-up | Use `Long2LongOpenHashMap` inside `PendingAcksMap` |
| 4 | Follow-up | Add the BitSet prefix index for same-ledger `removeAllUpTo` cleanup |
The O(1) `size()` change is already split separately in #26019. The current pending-ack storage PRs do not include the size change.
### Alternatives
A few alternatives were considered:
- Keep the current nested `TreeMap` layout. This preserves simple prefix cleanup, but keeps the per-entry object and boxing overhead.
- Copy or reimplement a primitive ordered tree map. This would preserve ordered-prefix operations, but the implementation cost and review risk are much higher.
- Add an external primitive collection dependency. This avoids local collection code, but introduces dependency and licensing/review overhead for a narrow broker hot-path use case.
The proposed approach keeps the ordered structure only where the semantics require it, and uses a smaller local primitive structure for the inner hot path.
### Anything else?
The draft PR and split PRs include tests for the changed assumptions:
- packed values preserve both `remainingUnacked` and `stickyKeyHash`;
- zero remaining counts are not confused with missing entries;
- negative sticky-key hashes round-trip correctly;
- whole-ledger and boundary-ledger cleanup remain correct;
- entries after the mark-delete position are retained;
- unordered inner storage still preserves `removeAllUpTo` behavior;
- BitSet fallback remains correct when the index cannot be used;
- the primitive map is checked with oracle, random, collision, and edge-case tests.
Benchmark and JOL data are included in comments. The benchmark uses datasets tied to real defaults such as `maxUnackedMessagesPerConsumer = 50000`, `maxUnackedMessagesPerSubscription = 200000`, and `receiverQueueSize = 1000`.
This does not change the ack protocol, subscription semantics, cursor mark-delete semantics, or dispatch/redelivery ordering. The goal is to reduce local data-structure overhead in the broker consumer pending-ack path.
### Are you willing to submit a PR?
- [x] I am willing to submit a PR!
Contributor guide
Research direction
Review draft PR #26024 and split PRs #26028 and #26030 first; the main entry points are PendingAcksMap and Consumer. Read the listed tests covering packed values, cleanup boundaries, unordered storage, and primitive-map edge cases. Done means preserving ack, mark-delete, and redelivery semantics while validating the stated memory and benchmark improvements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100