apache / apache/pulsar

[Bug] Partial batch ACKs are not scheduled for persistence when individual-delete ranges are empty

Open
#26,498 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I reproduced the issue on the `master` branch.

### User environment

- Version: `master` (`ce607988747`).
- Component: `managed-ledger`, `ManagedCursorImpl`.

### Issue Description

With batch-index acknowledgment enabled, acknowledging part of a batch can update the cursor in memory without scheduling persistence. The cursor's flush routine also skips the update, so the acknowledgment state can remain unsaved until another operation writes the cursor.

#### Triggering scenario

Assume a durable cursor has persisted its progress through entry E0, with no cursor write pending. E0 is its mark-delete position: all entries up to that position are acknowledged. The next entry, E1, contains two batched messages, A and B. A is acknowledged, but B is still pending.

Since A and B share the same stored entry, acknowledging A alone cannot mark E1 as fully deleted. Instead, the cursor updates a batch ACK bitmap. If there are no other fully acknowledged entries beyond E0, the whole-entry deletion ranges (`individualDeletedMessages`) remain empty even though the bitmap has changed.

`asyncDelete` treats the empty range collection as a reason to return early. It invokes `deleteComplete` but does not set `isDirty`. The periodic flush task calls the cursor's `flush()` routine, which returns immediately when that flag is false. The changed bitmap therefore has no pending write.

Expected: a changed batch ACK bitmap should be scheduled for persistence even when the mark-delete position and whole-entry deletion ranges are unchanged. Saving it should not depend on another acknowledgment or cursor close.

#### Impact

If the broker crashes before another operation persists this bitmap, the stored cursor state has no record of A's acknowledgment. A may therefore be delivered again after recovery. This is a risk of duplicate delivery, not loss of the message itself.

A later ACK that triggers persistence, an explicit mark-delete, or a graceful cursor close may save the accumulated bitmap, so the unsaved state is not necessarily visible after every restart.

### Reproducing the issue

These steps use the managed-ledger API on a durable `ManagedCursor` and inspect the cursor state written to its BookKeeper ledger.

1. In `ManagedLedgerConfig`, set `deletionAtBatchIndexLevelEnabled=true` and `throttleMarkDelete=0`. Disabling throttling rules out rate-limit delays; it is not a prerequisite for this bug, whose early return occurs before the rate-limit check.
2. Add entry E0, batch entry E1 containing A and B, and a later entry. Mark-delete E0 while the later entries remain unacknowledged, and wait for the cursor-state write to complete.
3. Individually acknowledge only A in E1. The in-memory batch bitmap changes, while the whole-entry deletion ranges remain empty.
4. Without another ACK or cursor close, invoke the cursor's `flush()`. The latest stored cursor record (`PositionInfo`) still has no batch ACK record for E1.
5. Mark-delete E0 again, without advancing the mark-delete position, and wait for completion. The stored record now contains E1's batch ACK bitmap.

The final step writes the bitmap without advancing the cursor, showing that the missing state can be serialized. The batch-record persistence limit is not reached in this scenario.

### Additional information

Relevant code: [partial-ACK update and early return](https://github.com/apache/pulsar/blob/ce6079887474043f4055ee6c22d2c72e456ac40f/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L2632-L2653), [flush dirty check](https://github.com/apache/pulsar/blob/ce6079887474043f4055ee6c22d2c72e456ac40f/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L4077-L4084).

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start in managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java at the partial-ACK early return around lines 2632-2653, then read the flush dirty check around lines 4077-4084. Reproduce the described E0/E1 scenario with batch-index acknowledgments and inspect the stored PositionInfo after flush; done means the changed batch ACK bitmap is persisted without another ACK or cursor close.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.