apache / apache/pulsar

Create producer issues using function worker as separate node

Open
#5,519 11 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
Hi,

Today, one of our zookeeper node had the following error:

```
Oct 31 10:01:06 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:06.697 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e569 zxid:0x450025f5be txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.316 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e575 zxid:0x450025f5cb txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.523 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d588 zxid:0x450025f5ce txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:35 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:35.465 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e58a zxid:0x450025f5d6 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:49 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:49.277 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d5ad zxid:0x450025f5e1 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:02:30 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:02:30.556 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e5b5 zxid:0x450025f5ea txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:04:08 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:04:08.683 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d605 zxid:0x450025f5f5 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
```
Then no log and zk node seems stucked. We restarted it and after a while (10 minutes), it recovered alone. WDYT?

Contributor guide

Open the contributing guide

Research direction

Start with the supplied ZooKeeper logs and the Pulsar function-worker/producer-node behavior described in the title. Trace why repeated “Directory not empty” errors leave the ZooKeeper node stuck, then verify the cause and document a tested fix or mitigation for the recovery behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.