No such ledger exists on Bookies but ledgermetadata exist

**BUG REPORT**

***Describe the bug***

We have noticed a strange behavior in our Bookkeeper cluster in production. In summary, we are currently unable to access the data of some ledgers that should have been created by Bookkeeper and therefore should exist. When we try to find the ledger using the Bookkeeper CLI:

```
./bookkeeper shell ledgermetadata -ledger 15543
24-04-14-12-30-18 ledgerID: 15543
24-04-14-12-30-18 LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2, ackQuorumSize=2, state=CLOSED, length=13417, lastEntryId=78, digestType=CRC32C, password=base64:, ensembles={0=[mn1-bookie2:1823, mn1-bookie3:1822]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:aWRjNzE3Ny9ucy9wZXJzaXN0ZW50L2V2ZW50cw==, application=base64:cHVsc2Fy}}
```
However, when we try to read the ledger using the CLI:
`./bookkeeper readledger -ledgerid 15543`

```
./bookkeeper readledger -ledgerid 15543
24-04-14-12-30-49 BookKeeper metadata driver manager initialized
24-04-14-12-30-49 Initialize zookeeper metadata driver at metadata service uri zk+hierarchical://mn1-clustermanager1.dna:1281;mn1-clustermanager2.dna:1281;mn1-clustermanager3.dna:1281/ledgers : zkServers = mn1-clustermanager1.dna:1281,mn1-clustermanager2.dna:1281,mn1-clustermanager3.dna:1281, ledgersRootPath = /ledgers.
24-04-14-12-30-49 Client environment:zookeeper.version=3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f, built on 2022-02-25 08:49 UTC
24-04-14-12-30-49 Client environment:host.name=mn1-bookie2
24-04-14-12-30-49 Client environment:java.version=19.0.2
24-04-14-12-30-49 Client environment:java.vendor=Eclipse Adoptium
24-04-14-12-30-49 Client environment:java.home=/usr/java/jdk-19.0.2+7
24-04-14-12-30-49 Client environment:java.class.path=/data/mn1/bookie/./.code/commons-codec-1.15.jar:/data/mn1/bookie/./.code/geoip2-4.0.1.jar:/data/mn1/bookie/./.code/pulsar-functions-runtime-3.0.3.jar:/data/mn1/bookie/./.code/lombok-1.18.30.jar:/data/mn1/bookie/./.code/zookeeper-jute-3.8.0.jar:/data/mn1/bookie/./.code/jetty-util-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/graal-sdk-22.0.0.jar:/data/mn1/bookie/./.code/netty-resolver-dns-native-macos-4.1.107.Final-osx-x86_64.jar:/data/mn1/bookie/./.code/commons-collections-3.2.2.jar:/data/mn1/bookie/./.code/javax.activation-api-1.2.0.jar:/data/mn1/bookie/./.code/hadoop-hdfs-client-2.10.2.jar:/data/mn1/bookie/./.code/ha-api-3.1.12.jar:/data/mn1/bookie/./.code/jaxb-api-2.3.1.jar:/data/mn1/bookie/./.code/batik-codec-1.17.jar:/data/mn1/bookie/./.code/netty-transport-sctp-4.1.107.Final.jar:/data/mn1/bookie/./.code/pngj-2.1.0.jar:/data/mn1/bookie/./.code/netty-tcnative-boringssl-static-2.0.65.Final-windows-x86_64.jar:/data/mn1/bookie/./.code/hbase-client-2.3.7.jar:/data/mn1/bookie/./.code/commons-digester-2.1.jar:/data/mn1/bookie/./.code/pulsar-metadata-3.0.3.jar:/data/mn1/bookie/./.code/batik-constants-1.17.jar:/data/mn1/bookie/./.code/simpleclient_tracer_common-0.16.0.jar:/data/mn1/bookie/./.code/RoaringBitmap-0.9.44.jar:/data/mn1/bookie/./.code/netty-transport-classes-epoll-4.1.107.Final.jar:/data/mn1/bookie/./.code/org.apache.oltu.oauth2.client-1.0.2.jar:/data/mn1/bookie/./.code/rocksdbjni-7.9.2.jar:/data/mn1/bookie/./.code/netty-tcnative-boringssl-static-2.0.65.Final-osx-x86_64.jar:/data/mn1/bookie/./.code/tomcat-util-scan-9.0.84.jar:/data/mn1/bookie/./.code/netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar:/data/mn1/bookie/./.code/jakarta.ws.rs-api-2.1.6.jar:/data/mn1/bookie/./.code/eclipse-collections-7.1.1.jar:/data/mn1/bookie/./.code/jetty-servlets-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/stax-ex-1.8.3.jar:/data/mn1/bookie/./.code/batik-anim-1.17.jar:/data/mn1/bookie/./.code/async-http-client-netty-utils-2.12.1.jar:/data/mn1/bookie/./.code/hbase-metrics-2.3.7.jar:/data/mn1/bookie/./.code/jsoup-1.15.3.jar:/data/mn1/bookie/./.code/netty-resolver-dns-native-macos-4.1.107.Final-osx-aarch_64.jar:/data/mn1/bookie/./.code/saaj-impl-1.5.2.jar:/data/mn1/bookie/./.code/hbase-shaded-netty-2.2.1.jar:/data/mn1/bookie/./.code/oshi-core-java11-6.4.0.jar:/data/mn1/bookie/./.code/netty-codec-http2-4.1.107.Final.jar:/data/mn1/bookie/./.code/core-3.0.1.jar:/data/mn1/bookie/./.code/jackson-databind-2.16.2.jar:/data/mn1/bookie/./.code/bookkeeper-common-4.16.4.jar:/data/mn1/bookie/./.code/jersey-entity-filtering-2.40.jar:/data/mn1/bookie/./.code/batik-util-1.17.jar:/data/mn1/bookie/./.code/bcpg-jdk18on-1.75.jar:/data/mn1/bookie/./.code/hbase-hadoop-compat-2.3.7.jar:/data/mn1/bookie/./.code/j2objc-annotations-1.3.jar:/data/mn1/bookie/./.code/javax.websocket-client-api-1.0.jar:/data/mn1/bookie/./.code/commons-lang-2.6.jar:/data/mn1/bookie/./.code/scripting-api-29.04.jar:/data/mn1/bookie/./.code/netty-tcnative-boringssl-static-2.0.65.Final-linux-aarch_64.jar:/data/mn1/bookie/./.code/winzipaes-1.0.1.jar:/data/mn1/bookie/./.code/jetty-alpn-server-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/protobuf-java-util-3.21.8.jar:/data/mn1/bookie/./.code/avatica-metrics-1.22.0.jar:/data/mn1/bookie/./.code/magnews.cache-29.04.jar:/data/mn1/bookie/./.code/jetty-servlet-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/hbase-hadoop2-compat-2.3.7.jar:/data/mn1/bookie/./.code/js-scriptengine-22.0.0.jar:/data/mn1/bookie/./.code/herddb-collections-0.29.0.jar:/data/mn1/bookie/./.code/pulsar-functions-worker-3.0.3.jar:/data/mn1/bookie/./.code/juel-impl-2.2.5.jar:/data/mn1/bookie/./.code/magnews.inbox.litmus-29.04.jar:/data/mn1/bookie/./.code/jakarta.mail-api-1.6.5.jar:/data/mn1/bookie/./.code/unbescape-1.1.3.RELEASE.jar:/data/mn1/bookie/./.code/xml-apis-1.4.01.jar:/data/mn1/bookie/./.code/asm-9.6.jar:/data/mn1/bookie/./.code/poi-ooxml-5.2.5.jar:/data/mn1/bookie/./.code/txw2-2.3.3.jar:/data/mn1/bookie/./.code/metrics-jmx-4.1.12.1.jar:/data/mn1/bookie/./.code/validation-api-1.1.0.Final.jar:/data/mn1/bookie/./.code/codahale-metrics-provider-4.16.4.jar:/data/mn1/bookie/./.code/slf4j-api-2.0.4.jar:/data/mn1/bookie/./.code/jfreechart-1.0.12.jar:/data/mn1/bookie/./.code/memory-0.8.3.jar:/data/mn1/bookie/./.code/failsafe-2.4.4.jar:/data/mn1/bookie/./.code/netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/data/mn1/bookie/./.code/pulsar-functions-secrets-3.0.3.jar:/data/mn1/bookie/./.code/javax.servlet-api-4.0.1.jar:/data/mn1/bookie/./.code/magnews-backend-json-29.04.jar:/data/mn1/bookie/./.code/magnews.inbox.core-29.04.jar:/data/mn1/bookie/./.code/gmbal-4.0.1.jar:/data/mn1/bookie/./.code/tomcat-catalina-9.0.84.jar:/data/mn1/bookie/./.code/netty-transport-udt-4.1.107.Final.jar:/data/mn1/bookie/./.code/managed-ledger-3.0.3.jar:/data/mn1/bookie/./.code/zookeeper-3.8.0.jar:/data/mn1/bookie/./.code/magnews.majordodo-29.04.jar:/data/mn1/bookie/./.code/herddb-net-0.29.0.jar:/data/mn1/bookie/./.code/jakarta.jws-api-2.1.0.jar:/data/mn1/bookie/./.code/hbase-metrics-api-2.3.7.jar:/data/mn1/bookie/./.code/imageio-jpeg-3.9.4.jar:/data/mn1/bookie/./.code/jetty-server-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/scrimage-core-4.1.1.jar:/data/mn1/bookie/./.code/audience-annotations-0.5.0.jar:/data/mn1/bookie/./.code/batik-xml-1.17.jar:/data/mn1/bookie/./.code/vertx-core-4.3.5.jar:/data/mn1/bookie/./.code/jcommon-1.0.15.jar:/data/mn1/bookie/./.code/magnews-xml-29.04.jar:/data/mn1/bookie/./.code/magnews.backup-29.04.jar:/data/mn1/bookie/./.code/jsr181-api-1.0-MR1.jar:/data/mn1/bookie/./.code/magnews.bookie-29.04.jar:/data/mn1/bookie/./.code/jtidy-r938.jar:/data/mn1/bookie/./.code/pulsar-package-filesystem-storage-3.0.3.jar:/data/mn1/bookie/./.code/jetty-continuation-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/bookkeeper-stats-api-4.16.4.jar:/data/mn1/bookie/./.code/jna-platform-jpms-5.12.1.jar:/data/mn1/bookie/./.code/hbase-shaded-miscellaneous-3.3.0.jar:/data/mn1/bookie/./.code/hk2-api-2.6.1.jar:/data/mn1/bookie/./.code/simpleclient_servlet-0.16.0.jar:/data/mn1/bookie/./.code/jna-jpms-5.12.1.jar:/data/mn1/bookie/./.code/magnews.linkchecker-29.04.jar:/data/mn1/bookie/./.code/jakarta.xml.bind-api-2.3.3.jar:/data/mn1/bookie/./.code/pulsar-common-3.0.3.jar:/data/mn1/bookie/./.code/magnews.elasticsearch-29.04.jar:/data/mn1/bookie/./.code/jersey-hk2-2.40.jar:/data/mn1/bookie/./.code/jctools-core-2.1.2.jar:/data/mn1/bookie/./.code/eclipse-collections-api-7.1.1.jar:/data/mn1/bookie/./.code/httpclient-4.5.13.jar:/data/mn1/bookie/./.code/magnews.tomcat.embedded-29.04.jar:/data/mn1/bookie/./.code/giotto.importer-29.04.jar:/data/mn1/bookie/./.code/elsa-3.0.0-M5.jar:/data/mn1/bookie/./.code/sshd-sftp-2.12.0.jar:/data/mn1/bookie/./.code/netty-tcnative-classes-2.0.65.Final.jar:/data/mn1/bookie/./.code/commons-lang3-3.11.jar:/data/mn1/bookie/./.code/jackson-core-asl-1.9.13.jar:/data/mn1/bookie/./.code/jetty-util-ajax-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/netty-codec-mqtt-4.1.107.Final.jar:/data/mn1/bookie/./.code/tomcat-jasper-9.0.84.jar:/data/mn1/bookie/./.code/pfl-basic-4.1.0.jar:/data/mn1/bookie/./.code/jackson-module-jsonSchema-2.16.2.jar:/data/mn1/bookie/./.code/magnews.spf-29.04.jar:/data/mn1/bookie/./.code/commons-collections4-4.4.jar:/data/mn1/bookie/./.code/pulsar-client-original-3.0.3.jar:/data/mn1/bookie/./.code/pulsar-client-admin-api-3.0.3.jar:/data/mn1/bookie/./.code/blazingcache-core-3.3.0.jar:/data/mn1/bookie/./.code/jackson-jaxrs-base-2.16.2.jar:/data/mn1/bookie/./.code/subethasmtp-5.2.7.jar:/data/mn1/bookie/./.code/jasperreports-mn-4.0.0.jar:/data/mn1/bookie/./.code/stringtemplate-3.2.1.jar:/data/mn1/bookie/./.code/jersey-common-2.40.jar:/data/mn1/bookie/./.code/netty-transport-rxtx-4.1.107.Final.jar:/data/mn1/bookie/./.code/junit-4.13.2.jar:/data/mn1/bookie/./.code/avro-1.11.3.jar:/data/mn1/bookie/./.code/herddb-mock-0.29.0.jar:/data/mn1/bookie/./.code/conscrypt-openjdk-uber-2.5.2.jar:/data/mn1/bookie/./.code/jersey-container-servlet-2.40.jar:/data/mn1/bookie/./.code/httpcore-4.4.13.jar:/data/mn1/bookie/./.code/sac-1.3.jar:/data/mn1/bookie/./.code/simpleclient_jetty-0.16.0.jar:/data/mn1/bookie/./.code/majordodo-client-0.17.0.jar:/data/mn1/bookie/./.code/jackson-datatype-jsr310-2.16.2.jar:/data/mn1/bookie/./.code/httpcore5-5.1.3.jar:/data/mn1/bookie/./.code/com-diennea-security-29.04.jar:/data/mn1/bookie/./.code/commons-compress-1.25.0.jar:/data/mn1/bookie/./.code/maxmind-db-3.0.0.jar:/data/mn1/bookie/./.code/common-image-3.9.4.jar:/data/mn1/bookie/./.code/hbase-protocol-shaded-2.3.7.jar:/data/mn1/bookie/./.code/perfmark-api-0.26.0.jar:/data/mn1/bookie/./.code/netty-transport-native-kqueue-4.1.107.Final-osx-x86_64.jar:/data/mn1/bookie/./.code/rxjava-3.0.1.jar:/data/mn1/bookie/./.code/jetty-client-9.4.54.v20240208.jar:/data/mn1/bookie/./.code/magnews.dsnparser-29.04.jar:/data/mn1/bookie/./.code/magnews.rss-29.04.jar:/data/mn1/bookie/./.code/magnews-util-12.13.0.jar:/data/mn1/bookie/./.code/netty-resolver-dns-classes-macos-4.1.107.Final.jar:/data/mn1/bookie/./.code/curator-client-5.1.0.jar:/data/mn1/bookie/./.code/tomcat-jsp-api-9.0.84.jar:/data/mn1/bookie/./.code/bookkeeper-server-4.16.4.jar:/data/mn1/bookie/./.code/pdfbox-2.0.28.jar:/data/mn1/bookie/./.code/stax2-api-4.2.2.jar:/data/mn1/bookie/./.code/typetools-0.5.0.jar:/data/mn1/bookie/./.code/httpcore5-h2-5.1.3.jar:/data/mn1/bookie/./.code/jackson-dataformat-yaml-2.16.2.jar:/data/mn1/bookie/./.code/customtables-29.04.jar:/data/mn1/bookie/./.code/jakarta.inject-2.6.1.jar:/data/mn1/bookie/./.code/netty-codec-haproxy-4.1.107.Final.jar:/data/mn1/bookie/./.code/rome-utils-2.1.0.jar:/data/mn1/bookie/./.code/itext-2.1.7.jar:/data/mn1/bookie/./.code/netty-codec-dns-4.1.107.Final.jar:/data/mn1/bookie/./.code/commons-io-2.15.1.jar:/data/mn1/bookie/./.code/ecj-3.26.0.jar:/data/mn1/bookie/./.code/barcode4j-2.1.jar:/data/mn1/bookie/./.code/bcutil-jdk18on-1.75.jar:/data/mn1/bookie/./.code/jna-5.12.1.jar:/data/mn1/bookie/./.code/netty-tcnative-boringssl-static-2.0.65.Final.jar:/data/mn1/bookie/./.code/netty-reactive-streams-2.0.6.jar:/data/mn1/bookie/./.code/openpdf-1.3.30.jar:/data/mn1/bookie/./.code/batik-i18n-1.17.jar:/data/mn1/bookie/./.code/curator-framework-5.1.0.jar:/data/mn1/bookie/./.code/jers
24-04-14-12-30-49 Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
24-04-14-12-30-49 Client environment:java.io.tmpdir=/tmp
24-04-14-12-30-49 Client environment:java.compiler=
24-04-14-12-30-49 Client environment:os.name=Linux
24-04-14-12-30-49 Client environment:os.arch=amd64
24-04-14-12-30-49 Client environment:os.version=5.4.17-2136.321.4.1.el8uek.x86_64
24-04-14-12-30-49 Client environment:user.name=magnews
24-04-14-12-30-49 Client environment:user.home=/home/magnews
24-04-14-12-30-49 Client environment:user.dir=/data/mn1/bookie
24-04-14-12-30-49 Client environment:os.memory.free=153MB
24-04-14-12-30-49 Client environment:os.memory.max=2906MB
24-04-14-12-30-49 Client environment:os.memory.total=186MB
24-04-14-12-30-49 Initiating client connection, connectString=mn1-clustermanager1.dna:1281,mn1-clustermanager2.dna:1281,mn1-clustermanager3.dna:1281 sessionTimeout=10000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@323b36e0
24-04-14-12-30-49 Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
24-04-14-12-30-49 jute.maxbuffer value is 1048575 Bytes
24-04-14-12-30-49 zookeeper.request.timeout value is 0. feature enabled=false
24-04-14-12-30-49 Opening socket connection to server mn1-clustermanager1.dna/10.200.86.86:1281.
24-04-14-12-30-49 SASL config status: Will not attempt to authenticate using SASL (unknown error)
24-04-14-12-30-49 Socket connection established, initiating session, client: /10.200.86.156:41888, server: mn1-clustermanager1.dna/10.200.86.86:1281
24-04-14-12-30-49 Session establishment complete on server mn1-clustermanager1.dna/10.200.86.86:1281, session id = 0x103c4496f7e0075, negotiated timeout = 10000
24-04-14-12-30-49 ZooKeeper client is connected now.
24-04-14-12-30-49 Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver because No network topology script is found when using script based DNS resolver.
24-04-14-12-30-49 Initialize rackaware ensemble placement policy @ @ /default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver.
24-04-14-12-30-49 Not weighted
24-04-14-12-30-49 Weighted ledger placement is not enabled
24-04-14-12-30-49 Update BookieInfoCache (writable bookie) mn1-bookie2:1823 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=1823, host=mn1-bookie2, protocol=bookie-rpc, auth=[], extensions=[]}]}
24-04-14-12-30-49 Update BookieInfoCache (writable bookie) mn1-bookie1:1822 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=1822, host=mn1-bookie1, protocol=bookie-rpc, auth=[], extensions=[]}]}
24-04-14-12-30-49 Update BookieInfoCache (writable bookie) mn1-bookie3:1822 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=1822, host=mn1-bookie3, protocol=bookie-rpc, auth=[], extensions=[]}]}
24-04-14-12-30-49 Adding a new node: /default-rack/mn1-bookie2:1823
24-04-14-12-30-49 Adding a new node: /default-rack/mn1-bookie1:1822
24-04-14-12-30-49 Adding a new node: /default-rack/mn1-bookie3:1822
24-04-14-12-30-49 Successfully connected to bookie: mn1-bookie2:1823 [id: 0xa0ee35b2, L:/10.200.86.156:37280 - R:mn1-bookie2/10.200.86.156:1823]
24-04-14-12-30-49 connection [id: 0xa0ee35b2, L:/10.200.86.156:37280 - R:mn1-bookie2/10.200.86.156:1823] authenticated as BookKeeperPrincipal{ANONYMOUS}
24-04-14-12-30-49 Successfully connected to bookie: mn1-bookie3:1822 [id: 0xe9a3189a, L:/10.200.86.156:38104 - R:mn1-bookie3/10.200.86.157:1822]
24-04-14-12-30-49 connection [id: 0xe9a3189a, L:/10.200.86.156:38104 - R:mn1-bookie3/10.200.86.157:1822] authenticated as BookKeeperPrincipal{ANONYMOUS}
24-04-14-12-30-49 Read of ledger entry failed: L15543 E0-E0, Sent to [mn1-bookie2:1823, mn1-bookie3:1822], Heard from [] : bitset = {}, Error = 'No such ledger exists on Bookies'. First unread entry is (-1, rc = null)
24-04-14-12-30-49 Error reading entry 0 from ledger 15543
24-04-14-12-30-49 org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83)
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncReadCallback.readComplete(SyncCallbackUtils.java:229)
at org.apache.bookkeeper.client.LedgerHandle$4.onFailure(LedgerHandle.java:818)
at org.apache.bookkeeper.common.concurrent.FutureEventListener.accept(FutureEventListener.java:38)
at org.apache.bookkeeper.common.concurrent.FutureEventListener.accept(FutureEventListener.java:26)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1589)

24-04-14-12-30-49 Closing the per channel bookie client for mn1-bookie2:1823
24-04-14-12-30-49 Closing the per channel bookie client for mn1-bookie3:1822
24-04-14-12-30-49 Disconnected from bookie channel [id: 0xa0ee35b2, L:/10.200.86.156:37280 ! R:mn1-bookie2/10.200.86.156:1823]
24-04-14-12-30-49 Disconnected from bookie channel [id: 0xe9a3189a, L:/10.200.86.156:38104 ! R:mn1-bookie3/10.200.86.157:1822]
24-04-14-12-30-49 The mainWorkerPool did not shutdown cleanly
24-04-14-12-30-49 An exception was thrown while closing send thread for session 0x103c4496f7e0075.
24-04-14-12-30-50 Session: 0x103c4496f7e0075 closed
24-04-14-12-30-50 Got an exception
24-04-14-12-30-50 com.google.common.util.concurrent.UncheckedExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
24-04-14-12-30-50 EventThread shut down for session: 0x103c4496f7e0075
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
at org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand.apply(ReadLedgerCommand.java:138)
at org.apache.bookkeeper.bookie.BookieShell$ReadLedgerEntriesCmd.runCmd(BookieShell.java:641)
at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:248)
at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2349)
at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2446)
Caused by: java.lang.RuntimeException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
at org.apache.bookkeeper.client.BookKeeperAdmin$LedgerEntriesIterator.hasNext(BookKeeperAdmin.java:457)
at org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand.readledger(ReadLedgerCommand.java:173)
at org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand.apply(ReadLedgerCommand.java:136)
... 4 more
Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83)
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncReadCallback.readComplete(SyncCallbackUtils.java:229)
at org.apache.bookkeeper.client.LedgerHandle$4.onFailure(LedgerHandle.java:818)
at org.apache.bookkeeper.common.concurrent.FutureEventListener.accept(FutureEventListener.java:38)
at org.apache.bookkeeper.common.concurrent.FutureEventListener.accept(FutureEventListener.java:26)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1589)

```

Note:

- We have not performed some maintenance on Bookkeeper's storage.
- The creation of the ledger was requested by Apache Pulsar.

We also checked in the entry log files, and it really seems that ledger does not exist.

Furthermore, when that ledger was created by Apache Pulsar, Pulsar did not give any errors during writing. But when trying to read the ledger, Bookkeeper responded with "No such ledger exists on Bookies."

Do you have any information on what the problem might be or how we can debug this issue?

***To Reproduce***

We were unable to reproduce the issue.

***Expected behavior***

Given that the metadata exists, I expect the ledger to actually exist on Bookkeeper as well. We have not performed any ledger deletions on Bookkeeper.

Pulsar version: 3.0.3
Bookeeper version: 4.16.4