apache / apache/iotdb

[Bug] 0.14.0版本,集群部署iotdb,不定时出现数据库无法操作的问题,报错Sink handle is aborted;Driver was interrupted

Open
#11,473 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

0.14.0

### Describe the bug and provide the minimal reproduce step

集群部署iotdb:1号机器:ConfigNode,DataNode;2号机器:DataNode;3号机器:DataNode
问题描述:不定时出现数据库无法操作的问题
3号机器的DataNode出现error log:
`2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)`

问题在重启集群的DataNode后解决,但仍然会不定时出现相同的问题

### What did you expect to see?

此问题的解决方法

### What did you see instead?

2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the 0.14.0 stack-trace entry points in Driver.java, SinkHandle.java, and FragmentInstanceExecution.java, then correlate the DataNode logs across the three-node cluster. Reproduce the intermittent failure if possible and trace why the sink is aborted and the driver interrupted. Done means the cause is identified and a verified fix or regression test prevents the failure without requiring a DataNode restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.