apache / apache/iotdb

[Bug] 0.14.0版本,集群部署iotdb,不定时出现数据库无法操作的问题,报错Sink handle is aborted;Driver was interrupted

Aperta
#11,473 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
6.4k
Fork
1.2k
Merge medio
1g 23h
PR unite (30g)
115

Descrizione

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

0.14.0

### Describe the bug and provide the minimal reproduce step

集群部署iotdb:1号机器:ConfigNode,DataNode;2号机器:DataNode;3号机器:DataNode
问题描述:不定时出现数据库无法操作的问题
3号机器的DataNode出现error log:
`2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)`

问题在重启集群的DataNode后解决,但仍然会不定时出现相同的问题

### What did you expect to see?

此问题的解决方法

### What did you see instead?

2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia dai punti di ingresso dello stack trace della versione 0.14.0 in Driver.java, SinkHandle.java e FragmentInstanceExecution.java, quindi correla i log di DataNode nel cluster a tre nodi. Riproduci l’errore intermittente, se possibile, e traccia il motivo per cui il sink viene interrotto e il driver viene interrotto. Il lavoro è completo quando la causa è stata identificata e un fix verificato o un test di regressione impedisce l’errore senza richiedere il riavvio di un DataNode.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
databases, distributed-systems
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
20/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.