[Bug] Closing ledger 165820 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
Broker version (check with bin/pulsar version)
pulsar-toolset-0:/pulsar$ bin/pulsar version
Current version of pulsar is: 4.0.6
Git Revision 4538ef7645c45a3c8686092128fde6c5d61c762b
Git Branch branch-4.0
Built by Lari Hotari on Laris-MBP.lan at 2025-07-30T13:37:25+0300
Broker Operating system and hardware type (check with uname -a on Unixes / systeminfo on Windows)
Linux pulsar-toolset-0 5.10.244-240.970.amzn2.x86_64 #1 SMP Tue Oct 7 09:40:03 UTC 2025 x86_64 GNU/Linux
Broker Java version (check with java -version)
pulsar-toolset-0:/pulsar$ java -version
openjdk version "21.0.8" 2025-07-15 LTS
OpenJDK Runtime Environment Corretto-21.0.8.9.1 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.8.9.1 (build 21.0.8+9-LTS, mixed mode)
### Issue Description
We observed that from client side with the Error message: System.TimeoutException: Could not send message to broker within given timeout. Then we find out some issues in the pulsar broker. Please see the ERROR messages below.
### Error messages
```text
2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Fencing exception on write: L165820 E41 on pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 165820 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read
2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L165820 E41
2025-10-27T19:02:50,854+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Creating a new ledger after closed 165820
2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Fencing exception on write: L165820 E41 on pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: []
2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 165820 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read
2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=165820
2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=165820,loopId=6e454574) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) [org.apache.bookkeeper-bookkeeper-server-4.17.2.jar:4.17.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) [org.apache.bookkeeper-bookkeeper-server-4.17.2.jar:4.17.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) [org.apache.zookeeper-zookeeper-3.9.3.jar:3.9.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:542) [org.apache.zookeeper-zookeeper-3.9.3.jar:3.9.3]
2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
2025-10-27T19:02:50,859+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
2025-10-27T19:02:50,863+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 165841
2025-10-27T19:02:50,863+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Created new ledger 165841
2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Error updating meta data with the new list of ledgers: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers/Research/TimepointConfiguration/persistent/DataEvents
2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Research/TimepointConfiguration/persistent/DataEvents Moving to Fenced state
2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Failed to update ledger list. z-node version mismatch. Closing managed ledger
2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://Research/TimepointConfiguration/DataEvents}, client=[id: 0xb4740df9, L:/100.64.10.130:6650 - R:/100.64.6.123:41984] [SR:100.64.6.123, state:Connected], producerName=pulsar-876-15695, producerId=29}, assignedBrokerLookupData: Optional.empty
```
### Reproducing the issue
This happened in our production environment, we haven't done any thing regarding to reproduce the issue.
### Additional information
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the ManagedLedgerImpl, PendingAddOp, LedgerHandle, MetadataUpdateLoop, and AbstractZkLedgerManager entries shown in the logs, tracing the ledger-fencing and z-node version-mismatch sequence. There is no reproduction or test named in the issue, so first establish a reproducible scenario; done should include a confirmed cause and regression coverage for the observed producer timeout and managed-ledger closure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100