apache / apache/pulsar

[Bug] broker: Stale backlog after creating Earliest subscription on trimmed topic

Open
#25,813 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

pulsar broker 3.0.12
java 17

### Issue Description

Observed on a 3.0.x based branch.

### What did you expect to see?

After the inactive subscription is deleted and all old ledgers are trimmed, recreating the durable subscription with `Earliest` should initialize the cursor with backlog `0` if there is no real remaining data.

`stats.backlog`, `stats-internal`, and precise backlog should be consistent.

### What did you see instead?

`admin topics stats` can report `backlog > 0`, while:

- `admin topics stats-internal` shows old ledgers already trimmed
- precise backlog is `0`
- the consumer cannot actually receive any message
- unloading the topic makes the reported backlog become `0`

### Error messages

```text

```

### Reproducing the issue

1. Create a partitioned topic.
2. Produce messages across multiple ledgers.
3. Configure retention aggressively so old ledgers are trimmed, leaving only a new empty current ledger.
5. Create a new and durable subscription name and `SubscriptionInitialPosition.Earliest`.
6. Compare:
- `admin topics stats `
- `admin topics stats-internal `
- precise backlog

### Additional information

Root cause is in `ManagedLedgerImpl#getFirstPositionAndCounter()` during earliest cursor initialization.

When all historical ledgers are already trimmed and only an empty current ledger remains, `getFirstPosition()` can return a synthetic position on the old last confirmed ledger (for example `469:-1`).
`getFirstPositionAndCounter()` then uses that synthetic position in `getNumberOfEntries(Range.openClosed(...))`, which counts entries from a ledger that no longer exists in `ledgers`.

This leads to an incorrect `messagesConsumedCounter` when a durable subscription is recreated with `SubscriptionInitialPosition.Earliest` after inactive subscription deletion. The inaccurate counter then propagates into approximate backlog stats.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at ManagedLedgerImpl#getFirstPositionAndCounter() and reproduce the scenario with a trimmed topic, a new durable Earliest subscription, and the admin topics stats commands. Compare stats, stats-internal, and precise backlog; done when an empty current ledger produces consistent zero backlog and the consumer receives no phantom messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.