apache / apache/pulsar

[improve][broker] Speed up PendingAcksMap ack tracking with primitive storage and 50%+ lower memory

Open
#26,027 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the issues and found nothing similar.

### Motivation

`PendingAcksMap` is maintained per consumer and is on the broker-side consumer ack path. It tracks pending acknowledgements by `(ledgerId, entryId)` and is touched by dispatch, individual ack, partial ack, redelivery, and mark-delete cleanup.

The current implementation uses nested ordered maps with boxed keys and object values. This keeps the code simple, but it also adds per-entry object overhead, allocation rate, and GC pressure when a consumer has a large pending-ack window.

This issue tracks a series of focused improvements to reduce that overhead without changing consumer ack semantics.

A full draft implementation is available for review and validation:

https://github.com/apache/pulsar/pull/26024

### Solution

The main insight is that `PendingAcksMap` needs ordering at the ledger level, but not for every entry within a ledger.

The outer ledger map must stay ordered because `removeAllUpTo(markDeletePosition)` removes whole ledger buckets by ledger range. Inside one ledger, the common operations are exact lookup, exact remove, remaining-unacked update, and boundary cleanup for `entryId <= markDeleteEntryId`. These operations do not require the inner map to iterate entries in sorted order.

Redelivery ordering is not inherited from the per-ledger pending-ack map. The redelivery path rebuilds ordered positions separately, so replacing the inner ordered map does not change the redelivery ordering contract.

The split is:

| Part | PR | Scope |
| --- | --- | --- |
| 1 | #26028 | Add local primitive `Long2LongOpenHashMap` |
| 2 | #26030 | Pack `PendingAcksMap` values into `long` and use primitive accessors in `Consumer` |
| 3 | Follow-up | Use `Long2LongOpenHashMap` inside `PendingAcksMap` |
| 4 | Follow-up | Add the BitSet prefix index for same-ledger `removeAllUpTo` cleanup |

The O(1) `size()` change is already split separately in #26019. The current pending-ack storage PRs do not include the size change.

### Alternatives

A few alternatives were considered:

- Keep the current nested `TreeMap` layout. This preserves simple prefix cleanup, but keeps the per-entry object and boxing overhead.
- Copy or reimplement a primitive ordered tree map. This would preserve ordered-prefix operations, but the implementation cost and review risk are much higher.
- Add an external primitive collection dependency. This avoids local collection code, but introduces dependency and licensing/review overhead for a narrow broker hot-path use case.

The proposed approach keeps the ordered structure only where the semantics require it, and uses a smaller local primitive structure for the inner hot path.

### Anything else?

The draft PR and split PRs include tests for the changed assumptions:

- packed values preserve both `remainingUnacked` and `stickyKeyHash`;
- zero remaining counts are not confused with missing entries;
- negative sticky-key hashes round-trip correctly;
- whole-ledger and boundary-ledger cleanup remain correct;
- entries after the mark-delete position are retained;
- unordered inner storage still preserves `removeAllUpTo` behavior;
- BitSet fallback remains correct when the index cannot be used;
- the primitive map is checked with oracle, random, collision, and edge-case tests.

Benchmark and JOL data are included in comments. The benchmark uses datasets tied to real defaults such as `maxUnackedMessagesPerConsumer = 50000`, `maxUnackedMessagesPerSubscription = 200000`, and `receiverQueueSize = 1000`.

This does not change the ack protocol, subscription semantics, cursor mark-delete semantics, or dispatch/redelivery ordering. The goal is to reduce local data-structure overhead in the broker consumer pending-ack path.

### Are you willing to submit a PR?

- [x] I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Review draft PR #26024 and split PRs #26028 and #26030 first; the main entry points are PendingAcksMap and Consumer. Read the listed tests covering packed values, cleanup boundaries, unordered storage, and primitive-map edge cases. Done means preserving ack, mark-delete, and redelivery semantics while validating the stated memory and benchmark improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.