[Bug] Bookie (v4.1.2) Restarting Suddenly: ZKRegistrationClient Invalidate Cache / NetworkTopology Node Removal
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
Pulsar Version: 4.1.2 (Image: docker.io/apachepulsar/pulsar-all:4.1.2)
Platform: Kubernetes (Azure/AKS based on log headers)
Component: BookKeeper / ZooKeeper
### Issue Description
Our pulsar-bookie pods are experiencing sudden restarts. The logs indicate that the ZKRegistrationClient is invalidating the cache for the specific bookie address, followed by NetworkTopologyImpl removing the node from the /default-rack.
### Error messages
```text
INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Invalidate cache for pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181
INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181
```
### Reproducing the issue
ZooKeeper logs show standard ruok commands but no explicit session expiration immediately preceding the drop.
The Bookie seems to be under normal load (Compaction usage buckets are mostly at 100%).
Config: diskUsageWarnThreshold = 0.9, isForceGCAllowWhenNoSpace = true.
### Additional information
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reproducing the restart on a supported Pulsar version and inspect the logs around ZKRegistrationClient and NetworkTopologyImpl. Compare the ZooKeeper session state with the bookie’s disk and compaction conditions. Done means identifying the cause of the cache invalidation and node removal, then adding a regression test and verified fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100