apache / apache/pulsar

[Bug] Topic Backlog not going to 0 after subscription is deleted.

Open
#24,686 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

Pulsar Versoin - 4.0.3
Broker Operating system - Linux
Java Version - 17

### Issue Description

One of our topics on our Geo-Replicated Cluster (without global zookeeper) is behaving oddly. The replication of the topic was stuck due to BacklogExceededException, as the producers could not message. When we looked into this, we found that the current backlog was twice the size of the backlog limit. Furthermore, the storage size is even larger than the backlog.

Therefore, we tried to delete the topic by setting the inactive topic policies, and removing the subscription under the topic. However, the topic was not deleted, its backlog and storage size (we have no retention set) also didn't change.

As the topic is replicated, we followed the documentation to use auto-delete of topic instead of manually doing it, but we aren't sure what went wrong with this topic.

All necessary properties of the topics are as follows -

**retention policies**

{
"retentionTimeInMinutes" : 0,
"retentionSizeInMB" : 0
}

**backlog quota policies are as follows:**

destination_storage BacklogQuotaImpl(limit=104857600, limitSize=104857600, limitTime=-1, policy=producer_request_hold)
message_age BacklogQuotaImpl(limit=104857600, limitSize=104857600, limitTime=-1, policy=producer_request_hold)

**Inactive Topic policies**
{
"inactiveTopicDeleteMode" : "delete_when_no_subscriptions",
"maxInactiveDurationSeconds" : 1,
"deleteWhileInactive" : true
}

**Topic Stats:**

{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 0,
"msgInCounter" : 0,
"systemTopicBytesInCounter" : 0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"bytesOutInternalCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 1118483562,
"backlogSize" : 257313549,
"backlogQuotaLimitSize" : 104857600,
"backlogQuotaLimitTime" : -1,
"oldestBacklogMessageAgeSeconds" : -1,
"publishRateLimitedTimes" : 0,
"earliestMsgPublishTimeInBacklogs" : 0,
"offloadedStorageSize" : 0,
"lastOffloadLedgerId" : 0,
"lastOffloadSuccessTimeStamp" : 0,
"lastOffloadFailureTimeStamp" : 0,
"ongoingTxnCount" : 0,
"abortedTxnCount" : 0,
"committedTxnCount" : 0,
"publishers" : [ ],
"waitingPublishers" : 0,
"subscriptions" : { },
"replication" : {
"cluster-a" : {
"msgRateIn" : 0.0,
"msgInCount" : 0,
"msgThroughputIn" : 0.0,
"bytesInCount" : 0,
"msgRateOut" : 0.0,
"msgOutCount" : 0,
"msgThroughputOut" : 0.0,
"bytesOutCount" : 0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 2488,
"connected" : false,
"replicationDelayInSeconds" : 0,
"msgExpiredCount" : 0
}
},
"deduplicationStatus" : "Disabled",
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 0,
"delayedMessageIndexSizeInBytes" : 0,
"compaction" : {
"lastCompactionRemovedEventCount" : 0,
"lastCompactionSucceedTimestamp" : 0,
"lastCompactionFailedTimestamp" : 0,
"lastCompactionDurationTimeInMills" : 0
},
"ownerBroker" : "<-REMOVED->"
}

**Internal Stats:**

{
"entriesAddedCounter" : 0,
"numberOfEntries" : 10815,
"totalSize" : 1118483562,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2025-08-28T15:01:17.784+02:00",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "304:10814",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 304,
"entries" : 10815,
"size" : 1118483562,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 312,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"pulsar.repl.cluster-a" : {
"markDeletePosition" : "304:8326",
"readPosition" : "304:8327",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -2488,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2025-08-28T15:01:17.795+02:00",
"state" : "NoLedger",
"active" : false,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : false,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ {
"ledgerId" : 300,
"entries" : 1,
"size" : 120,
"offloaded" : false,
"underReplicated" : false
} ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}

### Error messages

```text

```

### Reproducing the issue

1. Setup 2 pulsar cluster's cluster-a and cluster-b.
2. Enable Geo replication between the two clusters without global zookeeper.
3. Create a topic, enable bi-directional geo-replication on the topic.
4. Create a subscription sub-a on the topic on both clusters
5. Set replicated state of subscription to true
6. Start a producer and produce messages
7. Delete the subscription
8. Set inactive topic policies on topic to delete_when_no_subscriptions.

### Additional information

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the two-cluster, bidirectional geo-replication sequence described in the issue and inspect topic and internal stats after deleting the subscription. Done means the subscription deletion and inactive-topic policy allow the topic backlog and storage to reach zero or the topic to be deleted as configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.