apache / apache/bookkeeper

Bookie shutdown failed with fatal error

Open
#3,638 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**

***Describe the bug***

Bookie shutdown failed with fatal error. fatal error log is followed

```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=17742, tid=19134
#
# JRE version: Java(TM) SE Runtime Environment (17.0.5+9) (build 17.0.5+9-LTS-191)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0.5+9-LTS-191, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64)
# Problematic frame:
# C 0x0000000000000000
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/local/bin/core_pattern_pipe.py %e %p %t %u %g" (or dumping to /opt/pulsar/core.17742)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- S U M M A R Y ------------

Command Line: -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Dzookeeper.clientTcpKeepAlive=true -Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util.zip=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED --add-opens=java.base/sun.net=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Xms16g -Xmx16g -XX:MaxDirectMemorySize=48g -XX:MetaspaceSize=96m -XX:+UseZGC -XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch -Xlog:async -Xlog:gc*,safepoint:/opt/pulsar/logs/pulsar_bookie_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.maxCapacityPerThread=4096 -Dpulsar.log.appender=RoutingAppender -Dpulsar.log.dir=/opt/pulsar/logs -Dpulsar.log.level=info -Dpulsar.log.root.level=info -Dpulsar.routing.appender.default=Console -Dlog4j2.is.webapp=false -Dpulsar.functions.process.container.log.dir=/opt/pulsar/logs -Dpulsar.functions.java.instance.jar=/opt/pulsar/instances/java-instance.jar -Dpulsar.functions.python.instance.file=/opt/pulsar/instances/python-instance/python_instance_main.py -Dpulsar.functions.extra.dependencies.dir=/opt/pulsar/instances/deps -Dpulsar.functions.instance.classpath=/opt/pulsar/conf:::/opt/pulsar/lib/*: -Dpulsar.functions.log.conf=/opt/pulsar/conf/functions_log4j2.xml -Dbookkeeper.metadata.bookie.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataBookieDriver -Dbookkeeper.metadata.client.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataClientDriver -Dpulsar.log.file=bookkeeper.log org.apache.bookkeeper.server.Main --conf /opt/pulsar/conf/bookkeeper.conf

Host: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz, 64 cores, 251G, CentOS Linux release 7.4.1708 (Core)
Time: Sun Nov 13 23:24:13 2022 CST elapsed time: 404.939532 seconds (0d 0h 6m 44s)

--------------- T H R E A D ---------------

Current thread (0x00007f37d8000c80): JavaThread "qtp1258103238-126" [_thread_in_native, id=19134, stack(0x00007f3f250f3000,0x00007f3f251f4000)]

Stack: [0x00007f3f250f3000,0x00007f3f251f4000], sp=0x00007f3f251f1718, free space=1017k
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 7441 org.rocksdb.RocksDB.getLongProperty(JJLjava/lang/String;I)J (0 bytes) @ 0x00007f43957b7c88 [0x00007f43957b7c20+0x0000000000000068]
J 7440 c1 org.rocksdb.RocksDB.getLongProperty(Lorg/rocksdb/ColumnFamilyHandle;Ljava/lang/String;)J (26 bytes) @ 0x00007f438e81311c [0x00007f438e812f60+0x00000000000001bc]
J 7438 c1 org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.count()J (22 bytes) @ 0x00007f438e89384c [0x00007f438e893760+0x00000000000000ec]
J 7436 c1 org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex$$Lambda$165+0x0000000800e98220.get()Ljava/lang/Object; (8 bytes) @ 0x00007f438e3c651c [0x00007f438e3c6400+0x000000000000011c]
J 7229 c1 org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndexStats$1.getSample()Ljava/lang/Number; (5 bytes) @ 0x00007f438e92915c [0x00007f438e928fe0+0x000000000000017c]
J 7123 c2 org.apache.bookkeeper.stats.prometheus.SimpleGauge.getSample()Ljava/lang/Number; (10 bytes) @ 0x00007f43959adb04 [0x00007f43959adaa0+0x0000000000000064]
J 7069 c2 org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider$$Lambda$258+0x0000000800f65490.accept(Ljava/lang/Object;Ljava/lang/Object;)V (20 bytes) @ 0x00007f439599455c [0x00007f4395993b20+0x0000000000000a3c]
J 7039 c2 java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V java.base@17.0.5 (65 bytes) @ 0x00007f4395965220 [0x00007f4395964f80+0x00000000000002a0]
j org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.writeAllMetrics(Ljava/io/Writer;)V+27
j org.apache.bookkeeper.stats.prometheus.PrometheusServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+29
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+39
j org.eclipse.jetty.servlet.ServletHolder.handle(Lorg/eclipse/jetty/server/Request;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+25
j org.eclipse.jetty.servlet.ServletHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+180
j org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+27
j org.eclipse.jetty.server.handler.ContextHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+72
j org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+38
j org.eclipse.jetty.servlet.ServletHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+179
j org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+16
j org.eclipse.jetty.server.handler.ContextHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+382
j org.eclipse.jetty.server.handler.ScopedHandler.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+20
j org.eclipse.jetty.server.handler.HandlerWrapper.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+18
j org.eclipse.jetty.server.Server.handle(Lorg/eclipse/jetty/server/HttpChannel;)V+149
j org.eclipse.jetty.server.HttpChannel.lambda$handle$1()V+69
j org.eclipse.jetty.server.HttpChannel$$Lambda$257+0x0000000800f5ef70.dispatch()V+4
j org.eclipse.jetty.server.HttpChannel.dispatch(Ljavax/servlet/DispatcherType;Lorg/eclipse/jetty/server/HttpChannel$Dispatchable;)V+37
j org.eclipse.jetty.server.HttpChannel.handle()Z+220
j org.eclipse.jetty.server.HttpConnection.onFillable()V+120
j org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded()V+4
j org.eclipse.jetty.io.FillInterest.fillable()Z+57
j org.eclipse.jetty.io.ChannelEndPoint$1.run()V+7
j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(Ljava/lang/Runnable;)V+1
j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(Z)Z+507
j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(Z)V+134
j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run()V+2
j org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run()V+274
j org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Ljava/lang/Runnable;)V+1
j org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run()V+262
j java.lang.Thread.run()V+11 java.base@17.0.5
v ~StubRoutines::call_stub
```

***To Reproduce***
I don't know how to produce thie issue yet. but it does occur frequently during testing

***Expected behavior***
Shutdown with no error

***Screenshots***

If applicable, add screenshots to help explain your problem.

***Additional context***

Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.