iotdb 1.1.2 扩容失败 remove 扩容的节点 也不行
- Dominant language
- Java
- Stars
- 6.4k
- Forks
- 1.2k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 115
Description
logs/log_confignode_all.log 的报错
2023-11-23 16:30:56,189 [grpc-default-executor-0] WARN o.a.ratis.util.LogUtils:124 - 9: Failed APPEND_ENTRIES request 4->9#1-t4,previous=(t:2,
i:60),leaderCommit=34557,initializing? false,entries: size=34497, first=(t:3, i:61), CONFIGURATIONENTRY(current:id: "0"
address: "data-iotdb1:10720"
startupRole: FOLLOWER
,id: "2"
address: "data-iotdb2:10720"
startupRole: FOLLOWER
,id: "4"
address: "data-iotdb3:10720"
startupRole: FOLLOWER
, old:)
java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 9@group-000000000000 is not in [STARTING
, RUNNING]: current state is CLOSING
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:121)
at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$11(RaftServerImpl.java:825)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 9@group-000000000000 is not in [STARTING, RUNNING]: current state is CL
OSING
at org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:749)
at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
at org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:748)
at org.apache.ratis.server.impl.RaftServerImpl.preAppendEntriesAsync(RaftServerImpl.java:1449)
at org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1393)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$25(RaftServerProxy.java:637)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
... 5 common frames omitted
2023-11-23 16:30:56,204 [grpc-default-executor-1] WARN o.a.ratis.util.LogUtils:122 - 9: APPEND_ENTRIES onError, lastRequest: null: org.apache.r
atis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: client cancelled
remove-datanode.sh 报错如下
2023-11-23 17:12:04,799 [main] INFO o.a.i.d.s.DataNodeServerCommandLine:117 - Remove result TDataNodeRemoveResp(status:TSStatus(code:1003, message:Failed to remove all requested data nodes))
2023-11-23 17:12:04,801 [main] ERROR o.a.i.c.ServerCommandLine:63 - Failed to execute system command
org.apache.iotdb.commons.exception.IoTDBException: TSStatus(code:1003, message:Failed to remove all requested data nodes)
at org.apache.iotdb.db.service.DataNodeServerCommandLine.doRemoveDataNode(DataNodeServerCommandLine.java:120)
at org.apache.iotdb.db.service.DataNodeServerCommandLine.run(DataNodeServerCommandLine.java:83)
at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
at org.apache.iotdb.db.service.DataNode.main(DataNode.java:148)
Contributor guide
Research direction
Start with logs/log_confignode_all.log and the failure path at DataNodeServerCommandLine.java:120, then trace the expansion and remove-datanode workflow around the reported Ratis CLOSING state. Reproduce the failure on IoTDB 1.1.2 if possible. Done means the node expansion and subsequent removal complete successfully without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100