bookkeeper is running out of direct memory
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.8.2
### Minimal reproduce step
bookeeper runnning for some days and ran out of direct memory .
### What did you expect to see?
Bookkeeper should be running without any error .
we installed 2.8.2 and bookkeeper is not up and running .
We shall attach the heap dump form the bookie for the same .
### What did you see instead?
INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal data/bookkeeper/journal/current/181400c6508.txn
2022-11-14T22:01:22,123 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: bookie-server
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_332]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[?:1.8.0_332]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_332]
at [sun.nio.ch](http://sun.nio.ch/).Util.getTemporaryDirectBuffer(Util.java:247) ~[?:1.8.0_332]
at [sun.nio.ch](http://sun.nio.ch/).IOUtil.read(IOUtil.java:195) ~[?:1.8.0_332]
at [sun.nio.ch](http://sun.nio.ch/).FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_332]
at org.apache.bookkeeper.bookie.JournalChannel.read(JournalChannel.java:261) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Journal.fullRead(Journal.java:1194) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:833) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:995) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:961) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1015) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [com.google.guava-guava-30.1-jre.jar:?]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.Main.doMain(Main.java:234) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Calling uncaughtExceptionHandler
2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.ComponentStarter - Triggered exceptionHandler of Component: bookie-server because of Exception in Thread: Thread[main,5,main]
java.lang.OutOfMemoryError: Direct buffer memory
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with org.apache.bookkeeper.bookie.JournalChannel.read and the Journal.scanJournal/Bookie.replay path shown in the stack trace. Reproduce the BookKeeper 4.14.3 startup failure on the reported 2.8.2 installation and inspect any provided heap dump. Done means the direct-buffer exhaustion is explained and the bookie starts without the reported OutOfMemoryError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100