apache / apache/bookkeeper

BookKeeper 4.10.0 recovery issue

Open
#3,519 0 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**QUESTION**

BookKeeper 4.10.0 recovery issue.
A Bookie is broken beyond repair. When I try to restore a Ledger, I have this problem:
host3 is the broken host. When I try to recover it, it throws a ConnectTimeoutException with host3.

Am I using the recover command incorrectly?
`bin/bookkeeper shell recover host3:3181 -l 28190099`

This is the metadata of the ledger.

**ledgerID: 28190099
LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2, ackQuorumSize=2, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[ {{ host2 }} :3181, {{ host3 }}:3181]}, customMetadata={component=XX, pulsar/managed-ledger=XX, application=base64:XX}}**

`
11:30:14.419 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x4c920fe9, L:/{{ host1 }}:33605 - R:/{{ host2 }}:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
11:30:24.396 [bookkeeper-io-3-2] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x224fd5e3]/{{ host3 }}:3181, current state CONNECTING :
io.netty.channel.ConnectTimeoutException: connection timed out: /{{ host3 }}:3181
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:150) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
11:30:24.403 [BookKeeperClientWorker-OrderedExecutor-19-0] ERROR org.apache.bookkeeper.client.ReadLastConfirmedOp - While readLastConfirmed ledger: 28190099 did not hear success responses from all quorums
11:30:24.411 [BookKeeperClientWorker-OrderedExecutor-19-0] ERROR org.apache.bookkeeper.client.BookKeeperAdmin - BK error opening ledger: 28190099
org.apache.bookkeeper.client.BKException$BKReadException: Error while reading ledger
at org.apache.bookkeeper.client.BKException.create(BKException.java:62) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.BookKeeperAdmin$6.openComplete(BookKeeperAdmin.java:751) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.LedgerOpenOp$2.readLastConfirmedComplete(LedgerOpenOp.java:215) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.LedgerHandle$10.readLastConfirmedDataComplete(LedgerHandle.java:1405) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:142) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieClientImpl$2.safeRun(BookieClientImpl.java:369) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
11:30:24.412 [BookKeeperClientWorker-OrderedExecutor-19-0] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Recover bookie for 28190099 completed with rc : ReadException: Error while reading ledger
11:30:24.412 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for {{ host2 }}:3181
11:30:24.416 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for {{ host3 }}:3181
11:30:24.417 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x4c920fe9, L:/{{ host1 }}:33605 ! R:/{{ host2 }}:3181]
11:30:24.530 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x5826bcd8e8f081c closed
11:30:24.530 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x5826bcd8e8f081c
Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: Error while reading ledger
at org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.apply(RecoverCommand.java:110)
at org.apache.bookkeeper.bookie.BookieShell$RecoverCmd.runCmd(BookieShell.java:509)
at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:223)
at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:1976)
at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2067)
Caused by: org.apache.bookkeeper.client.BKException$BKReadException: Error while reading ledger
at org.apache.bookkeeper.client.BKException.create(BKException.java:62)
at org.apache.bookkeeper.client.BookKeeperAdmin.recoverBookieData(BookKeeperAdmin.java:613)
at org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.bkRecoveryLedger(RecoverCommand.java:213)
at org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.recover(RecoverCommand.java:155)
at org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.apply(RecoverCommand.java:108)
... 4 more
`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.