apache / apache/bookkeeper

How to recover from "IOException: Missing ledger signature" error?

Open
#2,485 1 comment 0 reactions 0 assignees View on GitHub
triage/week-50 type/question
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**QUESTION**

We have done a set of experiments with restarting a Kubernetes node with Bookies running inside. In one of the experiment, we found that one Bookie was not able to restart which the following:
```
2020-11-11 07:43:42,429 - INFO - [main:JournalChannel@154] - Opening journal /bk/journal/j1/current/175b5dae741.txn
2020-11-11 07:43:42,475 - ERROR - [main:Bookie@924] - Exception while replaying journals, shutting down
java.io.IOException: Missing ledger signature while reading header for /bk/index/current/1/9/109.idx
at org.apache.bookkeeper.bookie.FileInfo.readHeader(FileInfo.java:224)
at org.apache.bookkeeper.bookie.FileInfo.checkOpen(FileInfo.java:310)
at org.apache.bookkeeper.bookie.FileInfo.checkOpen(FileInfo.java:278)
at org.apache.bookkeeper.bookie.FileInfo.size(FileInfo.java:388)
at org.apache.bookkeeper.bookie.IndexPersistenceMgr.updatePage(IndexPersistenceMgr.java:643)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.grabLedgerEntryPage(IndexInMemPageMgr.java:447)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.getLedgerEntryPage(IndexInMemPageMgr.java:412)
at org.apache.bookkeeper.bookie.IndexInMemPageMgr.putEntryOffset(IndexInMemPageMgr.java:571)
at org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:103)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:530)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:512)
at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:366)
at org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:153)
at org.apache.bookkeeper.bookie.Bookie$4.process(Bookie.java:888)
at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:821)
at org.apache.bookkeeper.bookie.Journal.replay(Journal.java:866)
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:901)
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:922)
at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:141)
at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:58)
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:78)
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408)
at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113)
at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:80)
at org.apache.bookkeeper.server.Main.doMain(Main.java:229)
at org.apache.bookkeeper.server.Main.main(Main.java:203)
2020-11-11 07:43:42,489 - INFO - [main:ZooKeeper@693] - Session: 0x10007c8a7a803d8 closed
2020-11-11 07:43:42,490 - INFO - [main-EventThread:ClientCnxn$EventThread@522] - EventThread shut down for session: 0x10007c8a7a803d8
2020-11-11 07:43:42,546 - INFO - [vert.x-eventloop-thread-0:VertxHttpServer$2@79] - Starting Vertx HTTP server on port 8080
```
After this, the Bookie process starts but it is unable to do any IO, including the readiness probes of the [Bookkeeper Operator](https://github.com/pravega/bookkeeper-operator). The questions regarding this problem are the following:
- Is this `Missing ledger signature` expected in the presence when rebooting the node in which a Bookie runs? It looks as a form of data corruption/loss, but I would like to hear the confirmation from the Bookkeeper community about this.
- Once in the presence of this error (o similar ones), what is the best course of action? Should we decommission the Bookie and create a new one? I would like to hear what is the best approach to handle this situation.

Thanks in advance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.