apache / apache/pulsar

backlog quota is not reflecting on topic level

Open
#19,267 7 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar version : 2.8.2
I am using below code to set backlog quota on topic level.
BacklogQuota quota= BacklogQuota.builder().limitSize(sizeLimit).retentionPolicy(BacklogQuota.RetentionPolicy.producer_exception).build();
pulsarAdmin.topics().setBacklogQuota("testtopic",quota);

In broker log Its showing Its successfully set but If I am trying to get the backlog quota value ,Its not showing .This behavior is not consistent , Once It shows for 1 topic , going forward for other topics its not getting set .

### Minimal reproduce step

Crete multiple topics then try to set topic level backlog quota for each newly created topics .

### What did you expect to see?

In broker log Its saying successfully set whereas in get-backlog-quotas shows empty. So the backlog quota is not going to honor for topic.

**broker log:**
2023-01-17T14:22:10,927 [pulsar-io-4-6] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [itom-di-message-bus-svc] Successfully updated backlog quota map: namespace=public/default, topic=demo7, map={"destination_storage":{"limit":349525,"limitSize":349525,"limitTime":-1,"policy":"producer_exception"}}
2023-01-17T14:22:10,928 [pulsar-io-4-7] INFO org.apache.pulsar.broker.service.ServerCnx - [SystemTopic{topic=persistent://public/default/__change_events}][itomdipulsar-2-831] Closing producer on cnx /172.16.2.92:39646. producerId=306
2023-01-17T14:22:10,928 [pulsar-io-4-7] INFO org.apache.pulsar.broker.service.ServerCnx - [SystemTopic{topic=persistent://public/default/__change_events}][itomdipulsar-2-831] Closed producer on cnx /172.16.2.92:39646. producerId=306
2023-01-17T14:22:10,928 [pulsar-io-4-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/__change_events] [itomdipulsar-2-831] Closed Producer
2023-01-17T14:22:11,281 [pulsar-2-4] INFO org.apache.pulsar.broker.systopic.NamespaceEventsSystemTopicFactory - Create topic policies system topic client persistent://public/default/__change_events
2023-01-17T14:22:11,308 [pulsar-io-4-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/__change_events] [null] Creating producer on cnx [id: 0xc16d3c33, L:/172.16.2.92:39646 - R:172.16.2.92/172.16.2.92:6651]
2023-01-17T14:22:11,309 [pulsar-io-4-7] INFO org.apache.pulsar.broker.service.ServerCnx - [/172.16.2.92:39646][persistent://public/default/__change_events] Creating producer. producerId=307
2023-01-17T14:22:11,319 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/172.16.2.92:39646] Created new producer: Producer{topic=SystemTopic{topic=persistent://public/default/__change_events}, client=/172.16.2.92:39646, producerName=itomdipulsar-2-832, producerId=307}
2023-01-17T14:22:11,320 [pulsar-io-4-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/__change_events] [itomdipulsar-2-832] Created producer on cnx [id: 0xc16d3c33, L:/172.16.2.92:39646 - R:172.16.2.92/172.16.2.92:6651]
2023-01-17T14:22:11,322 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/__change_events] Creating a new ledger
2023-01-17T14:22:11,322 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/__change_events] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 100, 101, 102, 97, 117, 108, 116, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 95, 95, 99, 104, 97, 110, 103, 101, 95, 101, 118, 101, 110, 116, 115], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
2023-01-17T14:22:11,338 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.core-helm.svc.cluster.local.:3181, itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.core-helm.svc.cluster.local.:3181] for ledger: 250
2023-01-17T14:22:11,339 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/__change_events] Created new ledger 250
2023-01-17T14:22:11,349 [pulsar-ordered-OrderedExecutor-2-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000 sessionid:0x300294417740015 local:/172.16.2.92:49624 remoteserver:itomdipulsar-zookeeper/172.16.2.93:2281 lastZxid:4294970959 xid:13128 sent:13128 recv:13244 queuedpkts:1 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/managed-ledgers/public/default/persistent/__change_events
2023-01-17T14:22:11,350 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Reset cursor:ManagedCursorImpl{ledger=public/default/persistent/__change_events, name=__compaction, ackPos=195:18, readPos=195:19} to 200:-1 since ledger consumed completely
2023-01-17T14:22:11,351 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Reset cursor:ManagedCursorImpl{ledger=public/default/persistent/__change_events, name=pulsar.dedup, ackPos=195:18, readPos=195:19} to 200:-1 since ledger consumed completely
2023-01-17T14:22:11,351 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Cursor: NonDurableCursorImpl{ledger=public/default/persistent/__change_events, ackPos=134:2, readPos=200:1} does not exist in the managed-ledger.

### What did you see instead?

get-backlog-quotas shouldn't be empty and backlog quota is should honor for topics.

### Anything else?

Attached system topic stats.
[system_topic_stats.zip](https://github.com/apache/pulsar/files/10442050/system_topic_stats.zip)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the pulsarAdmin.topics().setBacklogQuota and get-backlog-quotas entry points, then inspect the PersistentTopicsBase update path mentioned in the broker log. Reproduce the behavior across multiple newly created topics and compare the stored quota with the successful update message; done means each topic reports and honors its configured backlog quota.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.