apache / apache/iotdb

iotdb 1.1.2 扩容失败 remove 扩容的节点 也不行

Open
#11,608 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

logs/log_confignode_all.log 的报错
2023-11-23 16:30:56,189 [grpc-default-executor-0] WARN o.a.ratis.util.LogUtils:124 - 9: Failed APPEND_ENTRIES request 4->9#1-t4,previous=(t:2,
i:60),leaderCommit=34557,initializing? false,entries: size=34497, first=(t:3, i:61), CONFIGURATIONENTRY(current:id: "0"
address: "data-iotdb1:10720"
startupRole: FOLLOWER
,id: "2"
address: "data-iotdb2:10720"
startupRole: FOLLOWER
,id: "4"
address: "data-iotdb3:10720"
startupRole: FOLLOWER
, old:)
java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 9@group-000000000000 is not in [STARTING
, RUNNING]: current state is CLOSING
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:121)
at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$11(RaftServerImpl.java:825)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 9@group-000000000000 is not in [STARTING, RUNNING]: current state is CL
OSING
at org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:749)
at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
at org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:748)
at org.apache.ratis.server.impl.RaftServerImpl.preAppendEntriesAsync(RaftServerImpl.java:1449)
at org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1393)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$25(RaftServerProxy.java:637)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
... 5 common frames omitted
2023-11-23 16:30:56,204 [grpc-default-executor-1] WARN o.a.ratis.util.LogUtils:122 - 9: APPEND_ENTRIES onError, lastRequest: null: org.apache.r
atis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: client cancelled

remove-datanode.sh 报错如下
2023-11-23 17:12:04,799 [main] INFO o.a.i.d.s.DataNodeServerCommandLine:117 - Remove result TDataNodeRemoveResp(status:TSStatus(code:1003, message:Failed to remove all requested data nodes))
2023-11-23 17:12:04,801 [main] ERROR o.a.i.c.ServerCommandLine:63 - Failed to execute system command
org.apache.iotdb.commons.exception.IoTDBException: TSStatus(code:1003, message:Failed to remove all requested data nodes)
at org.apache.iotdb.db.service.DataNodeServerCommandLine.doRemoveDataNode(DataNodeServerCommandLine.java:120)
at org.apache.iotdb.db.service.DataNodeServerCommandLine.run(DataNodeServerCommandLine.java:83)
at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
at org.apache.iotdb.db.service.DataNode.main(DataNode.java:148)

Contributor guide

Open the contributing guide

Research direction

Start with logs/log_confignode_all.log and the failure path at DataNodeServerCommandLine.java:120, then trace the expansion and remove-datanode workflow around the reported Ratis CLOSING state. Reproduce the failure on IoTDB 1.1.2 if possible. Done means the node expansion and subsequent removal complete successfully without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.