How to recover from "IOException: Missing ledger signature" error?
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
**QUESTION**
We have done a set of experiments with restarting a Kubernetes node with Bookies running inside. In one of the experiment, we found that one Bookie was not able to restart which the following:
```
2020-11-11 07:43:42,429 - INFO - [main:JournalChannel@154] - Opening journal /bk/journal/j1/current/175b5dae741.txn
2020-11-11 07:43:42,475 - ERROR - [main:Bookie@924] - Exception while replaying journals, shutting down
java.io.IOException: Missing ledger signature while reading header for /bk/index/current/1/9/109.idx
at org.apache.bookkeeper.bookie.FileInfo.readHeader(FileInfo.java:224)
at org.apache.bookkeeper.bookie.FileInfo.checkOpen(FileInfo.java:310)
at org.apache.bookkeeper.bookie.FileInfo.checkOpen(FileInfo.java:278)
at org.apache.bookkeeper.bookie.FileInfo.size(FileInfo.java:388)
at org.apache.bookkeeper.bookie.IndexPersistenceMgr.updatePage(IndexPersistenceMgr.java:643)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.grabLedgerEntryPage(IndexInMemPageMgr.java:447)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.getLedgerEntryPage(IndexInMemPageMgr.java:412)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.putEntryOffset(IndexInMemPageMgr.java:571)
at org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:103)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:530)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:512)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:366)
at org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:153)
at org.apache.bookkeeper.bookie.Bookie$4.process(Bookie.java:888)
at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:821)
at org.apache.bookkeeper.bookie.Journal.replay(Journal.java:866)
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:901)
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:922)
at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:141)
at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:58)
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:78)
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408)
at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113)
at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:80)
at org.apache.bookkeeper.server.Main.doMain(Main.java:229)
at org.apache.bookkeeper.server.Main.main(Main.java:203)
2020-11-11 07:43:42,489 - INFO - [main:ZooKeeper@693] - Session: 0x10007c8a7a803d8 closed
2020-11-11 07:43:42,490 - INFO - [main-EventThread:ClientCnxn$EventThread@522] - EventThread shut down for session: 0x10007c8a7a803d8
2020-11-11 07:43:42,546 - INFO - [vert.x-eventloop-thread-0:VertxHttpServer$2@79] - Starting Vertx HTTP server on port 8080
```
After this, the Bookie process starts but it is unable to do any IO, including the readiness probes of the [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator). The questions regarding this problem are the following:
- Is this `Missing ledger signature` expected in the presence when rebooting the node in which a Bookie runs? It looks as a form of data corruption/loss, but I would like to hear the confirmation from the Bookkeeper community about this.
- Once in the presence of this error (o similar ones), what is the best course of action? Should we decommission the Bookie and create a new one? I would like to hear what is the best approach to handle this situation.
Thanks in advance.
Contributor guide
Assessment
This issue has not been assessed yet.