apache / apache/pulsar

[Bug] Namespace is being unloaded

Open
#18,523 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar 2.9.2
Kubernetes v1.22.15
The topic is a partitioned, persistent topic with a deduplication enabled.

### Minimal reproduce step

Cannot reproduce, a k8 cluster was just restarted as usual, and some services redeployed, but pulsar stopped allowing connecting the producer.

### What did you expect to see?

Connecting producer to the topic.

### What did you see instead?

**In the client log:**
...
ERROR o.a.pulsar.client.impl.ProducerImpl - [persistent://datalake/ingress/real_time_doc_stream-partition-51] [i03-pulsar-76-2782] Failed to create producer: Namespace is being unloaded, cannot add topic persistent://datalake/ingress/real_time_doc_stream-partition-51
...
**In the broker log** (every 5sec)
...
14:11:55.095 [pulsar-io-4-9] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [i03-pulsar-bookie-2.i03-pulsar-bookie.str-i03.svc.cluster.local:3181, i03-pulsar-bookie-0.i03-pulsar-bookie.str-i03.svc
││ i03-pulsar-broker 14:12:02.613 [deduplication-snapshot-monitor-37-1] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
││ i03-pulsar-broker java.lang.NullPointerException: null
││ i03-pulsar-broker at org.apache.pulsar.broker.service.persistent.MessageDeduplication.takeSnapshot(MessageDeduplication.java:520) ~[org.apache.pulsar-pulsar-broker-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkDeduplicationSnapshot(PersistentTopic.java:2375) ~[org.apache.pulsar-pulsar-broker-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
││ i03-pulsar-broker at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$77(BrokerService.java:1682) ~[org.apache.pulsar-pulsar-broker-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1680) ~[org.apache.pulsar-pulsar-broker-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.pulsar.broker.service.BrokerService.lambda$startDeduplicationSnapshotMonitor$4(BrokerService.java:484) ~[org.apache.pulsar-pulsar-broker-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[org.apache.pulsar-managed-ledger-2.9.2.jar:2.9.2]
││ i03-pulsar-broker at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
││ i03-pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
││ i03-pulsar-broker at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
││ i03-pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
││ i03-pulsar-broker at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
││ i03-pulsar-broker at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
││ i03-pulsar-broker at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
││ i03-pulsar-broker at java.lang.Thread.run(Thread.java:829) ~[?:?]

### Anything else?

With pulsar-admin, tried to:
* unload a namecace
* restart brokers
* clear the backlog of the namespace
without result, a producer cannot connect due to a Namespace being in unloading state.

Looks like a namespace is stale in some unload process with some MessageDeduplication.takeSnapshot NPE.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with MessageDeduplication.takeSnapshot at line 520 and PersistentTopic.checkDeduplicationSnapshot, then trace the BrokerService deduplication snapshot monitor and namespace unload path described in the logs. The report cannot reproduce the failure, so use the supplied stack trace and broker logs to identify the interaction. Done means the unloading state no longer persists after the failure and producers can connect again.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.