Create producer issues using function worker as separate node
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
Hi,
Today, one of our zookeeper node had the following error:
```
Oct 31 10:01:06 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:06.697 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e569 zxid:0x450025f5be txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.316 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e575 zxid:0x450025f5cb txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:13 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:13.523 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d588 zxid:0x450025f5ce txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:35 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:35.465 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e58a zxid:0x450025f5d6 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:01:49 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:01:49.277 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d5ad zxid:0x450025f5e1 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:02:30 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:02:30.556 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x1037b708532006e type:delete cxid:0x46e5b5 zxid:0x450025f5ea txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
Oct 31 10:04:08 clevercloud-zookeeper-c2-n3 pulsar-zookeeper[18494]: 10:04:08.683 [ProcessThread(sid:3 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x3037b7082ea00fa type:delete cxid:0x6d605 zxid:0x450025f5f5 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0008 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0008
```
Then no log and zk node seems stucked. We restarted it and after a while (10 minutes), it recovered alone. WDYT?
Contributor guide
Research direction
Start with the supplied ZooKeeper logs and the Pulsar function-worker/producer-node behavior described in the title. Trace why repeated “Directory not empty” errors leave the ZooKeeper node stuck, then verify the cause and document a tested fix or mitigation for the recovery behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100