HugeGraphServer1.0.0集群模式部署成功,stop后再start报错Failed to decompress snapshot tar java.lang.NullPointerException: null
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 636
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 14
Description
### Bug Type (问题类型)
server status (启动/运行异常)
### Before submit
- [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)
### Environment (环境信息)
- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB 3 nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: 6500万 vertices, 18亿 edges
### Expected & Actual behavior (期望与实际表现)
启动失败
Starting HugeGraphServer...
Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)......Starting HugeGraphServer failed
See /opt/hugegraph/hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output.
查看日志后报错Failed to decompress snapshot tar java.lang.NullPointerException: null
2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [INFO] o.a.h.b.s.r.StoreSnapshotFile - The snapshot tar files to be loaded are [snapshot_rocksdb-data.zip]
2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [ERROR] o.a.h.b.s.r.StoreSnapshotFile - Failed to decompress snapshot tar
java.lang.NullPointerException: null
at org.apache.hugegraph.backend.store.raft.StoreSnapshotFile.decompressSnapshot(StoreSnapshotFile.java:212) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.backend.store.raft.StoreSnapshotFile.load(StoreSnapshotFile.java:123) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.backend.store.raft.StoreStateMachine.onSnapshotLoad(StoreStateMachine.java:209) ~[hugegraph-core-1.0.0.jar:1.0.0]
at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:690) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) ~[jraft-core-1.3.11.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.NodeImpl - Node got error: Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>: StateMachine onSnapshotLoad failed]].
2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.FSMCallerImpl - FSMCaller already in error status, ignore new error.
com.alipay.sofa.jraft.error.RaftException: StateMachine onSnapshotLoad failed
at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:692) [jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) [jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) [jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) [jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) [jraft-core-1.3.11.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2023-05-24 18:07:39 [JRaft-FSMCaller-Disruptor-0] [ERROR] o.a.h.b.s.r.StoreStateMachine - Raft error: StateMachine onSnapshotLoad failed
com.alipay.sofa.jraft.error.RaftException: StateMachine onSnapshotLoad failed
at com.alipay.sofa.jraft.core.FSMCallerImpl.doSnapshotLoad(FSMCallerImpl.java:692) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:406) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl.access$100(FSMCallerImpl.java:72) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:149) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:141) ~[jraft-core-1.3.11.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:137) [disruptor-3.3.7.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2023-05-24 18:07:39 [main] [ERROR] c.a.s.j.s.s.SnapshotExecutorImpl - Fail to load snapshot from ./raft-log/snapshot, FirstSnapshotLoadDone status is Status[UNKNOWN<-1>: StateMachine onSnapshotLoad failed].
2023-05-24 18:07:39 [main] [ERROR] c.a.s.j.c.NodeImpl - Node initSnapshotStorage failed.
2023-05-24 18:07:39 [main] [INFO] o.a.h.StandardHugeGraph - Close graph standardhugegraph[hugegraph]
2023-05-24 18:07:39 [main] [INFO] o.a.h.b.s.r.RaftContext - Stop raft server: 192.168.30.31:8091
2023-05-24 18:07:39 [main] [ERROR] c.a.s.r.s.ServerFactory - RPC-020100018: 关闭 server 时出错,key=[8,091].
com.alipay.remoting.LifeCycleException: this component has closed
at com.alipay.remoting.AbstractLifeCycle.shutdown(AbstractLifeCycle.java:41) ~[bolt-1.6.4.jar:?]
at com.alipay.remoting.AbstractRemotingServer.shutdown(AbstractRemotingServer.java:110) ~[bolt-1.6.4.jar:?]
at com.alipay.remoting.AbstractRemotingServer.stop(AbstractRemotingServer.java:84) ~[bolt-1.6.4.jar:?]
at com.alipay.sofa.rpc.server.bolt.BoltServer.stop(BoltServer.java:166) ~[sofa-rpc-all-5.7.6.jar:5.7.6]
at com.alipay.sofa.rpc.server.bolt.BoltServer.destroy(BoltServer.java:228) ~[sofa-rpc-all-5.7.6.jar:5.7.6]
at com.alipay.sofa.rpc.server.ServerFactory.destroyServer(ServerFactory.java:159) [sofa-rpc-all-5.7.6.jar:5.7.6]
at com.alipay.sofa.rpc.config.ServerConfig.destroy(ServerConfig.java:245) [sofa-rpc-all-5.7.6.jar:5.7.6]
at org.apache.hugegraph.rpc.RpcServer.destroy(RpcServer.java:138) [hugegraph-rpc-1.0.0.jar:1.0.0]
at org.apache.hugegraph.core.GraphManager.destroyRpcServer(GraphManager.java:320) [hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.core.GraphManager.close(GraphManager.java:266) [hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:132) [hugegraph-api-1.0.0.jar:0.69.0.0]
at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) [jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) [jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) [jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) [jersey-container-grizzly2-http-3.0.3.jar:?]
at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) [grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) [grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) [grizzly-http-server-3.0.1.jar:3.0.1]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:71) [hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:178) [hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:32) [hugegraph-dist-1.0.0.jar:1.0.0]
at org.apache.hugegraph.dist.HugeGraphServer.(HugeGraphServer.java:60) [hugegraph-dist-1.0.0.jar:1.0.0]
at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:120) [hugegraph-dist-1.0.0.jar:1.0.0]
2023-05-24 18:07:39 [main] [INFO] o.a.h.r.RpcServer - RpcServer stop on port 8091
2023-05-24 18:07:39 [main] [ERROR] o.a.h.d.HugeGraphServer - HugeRestServer start error:
java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason.
at com.alipay.sofa.jraft.RaftServiceFactory.createAndInitRaftNode(RaftServiceFactory.java:48) ~[jraft-core-1.3.11.jar:?]
at com.alipay.sofa.jraft.RaftGroupService.start(RaftGroupService.java:129) ~[jraft-core-1.3.11.jar:?]
at org.apache.hugegraph.backend.store.raft.RaftNode.initRaftNode(RaftNode.java:279) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.backend.store.raft.RaftNode.(RaftNode.java:61) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.backend.store.raft.RaftContext.initRaftNode(RaftContext.java:156) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.backend.store.raft.RaftBackendStoreProvider.waitReady(RaftBackendStoreProvider.java:152) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.StandardHugeGraph.waitReady(StandardHugeGraph.java:318) ~[hugegraph-core-1.0.0.jar:1.0.0]
at org.apache.hugegraph.core.GraphManager.waitGraphsReady(GraphManager.java:387) ~[hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.core.GraphManager.init(GraphManager.java:111) ~[hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:130) ~[hugegraph-api-1.0.0.jar:0.69.0.0]
at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) ~[jersey-container-grizzly2-http-3.0.3.jar:?]
at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:71) ~[hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:178) ~[hugegraph-api-1.0.0.jar:0.69.0.0]
at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:32) ~[hugegraph-dist-1.0.0.jar:1.0.0]
at org.apache.hugegraph.dist.HugeGraphServer.(HugeGraphServer.java:60) ~[hugegraph-dist-1.0.0.jar:1.0.0]
at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:120) ~[hugegraph-dist-1.0.0.jar:1.0.0]
2023-05-24 18:07:39 [main] [INFO] o.a.h.HugeFactory - HugeFactory shutdown
删除hugeserver目录下
rm -rf raft-log rocksdb-data/
然后重新初始化
启动hugeserver能启动成功
### Vertex/Edge example (问题点 / 边数据举例)
_No response_
### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.