apache / apache/iotdb

[Bug] 0.14.0版本,集群部署iotdb,不定时出现数据库无法操作的问题,报错Sink handle is aborted;Driver was interrupted

Abierto
#11,473 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
6.4k
Forks
1.2k
Merge medio
1 d 23 h
PR fusionados (30 d)
115

Descripción

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

0.14.0

### Describe the bug and provide the minimal reproduce step

集群部署iotdb:1号机器:ConfigNode,DataNode;2号机器:DataNode;3号机器:DataNode
问题描述:不定时出现数据库无法操作的问题
3号机器的DataNode出现error log:
`2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)`

问题在重启集群的DataNode后解决,但仍然会不定时出现相同的问题

### What did you expect to see?

此问题的解决方法

### What did you see instead?

2023-11-03 09:59:55,593 [Query-Worker-Thread-10$20231103_015954_06075_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
2023-11-03 09:59:55,882 [Query-Worker-Thread-10$20231103_015954_06262_1.2.0] ERROR o.a.i.d.m.e.s.AbstractDriverThread:76 - [ExecuteFailed]
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:204)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:125)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:270)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:118)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:70)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.checkState(SinkHandle.java:351)
at org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.isFull(SinkHandle.java:132)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:181)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:440)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:148)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:120)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con los puntos de entrada del stack trace de la versión 0.14.0 en Driver.java, SinkHandle.java y FragmentInstanceExecution.java; después, correlaciona los logs de DataNode en el clúster de tres nodos. Reproduce el fallo intermitente si es posible y rastrea por qué se aborta el sink y se interrumpe el driver. La tarea está completa cuando se identifica la causa y un fix verificado o una prueba de regresión evita el fallo sin requerir reiniciar un DataNode.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
databases, distributed-systems
Tipo de issue
Error
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
20/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.