apache / apache/iotdb

[Bug] 0.14.0版本,集群部署iotdb,不定时出现数据库无法操作的问题,报错Sink handle is aborted;Driver was interrupted

Offen
#11,473 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
6.4k
Forks
1.2k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
115

Beschreibung

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

0.14.0

### Describe the bug and provide the minimal reproduce step

集群部署iotdb:1号机器:ConfigNode,DataNode;2号机器:DataNode;3号机器:DataNode
问题描述:不定时出现数据库无法操作的问题
3号机器的DataNode出现error log:
`2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)`

问题在重启集群的DataNode后解决,但仍然会不定时出现相同的问题

### What did you expect to see?

此问题的解决方法

### What did you see instead?

2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit den Stack-Trace-Einstiegspunkten der Version 0.14.0 in Driver.java, SinkHandle.java und FragmentInstanceExecution.java und korrelieren Sie anschließend die DataNode-Logs im Drei-Knoten-Cluster. Reproduzieren Sie den intermittierenden Fehler, falls möglich, und verfolgen Sie nach, warum der sink abgebrochen und der driver unterbrochen wird. Die Aufgabe ist abgeschlossen, wenn die Ursache identifiziert ist und ein verifizierter Fix oder Regressionstest den Fehler verhindert, ohne einen Neustart eines DataNode zu erfordern.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
databases, distributed-systems
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
20/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.