三节点集群出现某个节点查询时间超长异常
- Lingua principale
- Java
- Stelle
- 6.4k
- Fork
- 1.2k
- Merge medio
- 1g 23h
- PR unite (30g)
- 115
Descrizione
三节点集群使用过程中,同样的查询sql,两个节点查询时间几十毫秒,另外一个节点需要80多秒
1.三个节点1,2,3在集群搭建完使用过程中,五六天前第3节点服务器重启过,重启后重启了第3节点的iotdb服务
2.再之后间隔了一天左右1,2节点服务器也重启过,同样重启了两节点的iotdb服务,重启后执行过sql数据和响应时间是正常的
3.再之后过元旦,今天发现第3节点查询数据时间超级慢,1、2节点查询响应很快是正常的
4.然后尝试了重启第3节点的iotdb服务,重启之后第3节点还是很慢为80多秒
5.查看了很慢节点的log_all的日志文件发现日志描述Failed to synchronize with the leader after #20004ms执行了四次加起来差不多是80多秒,不知道有没有遇到同样的问题的人或懂得大神能给解解惑
log_all的日志
2022-01-04 09:46:37,223 [ClusterClient-1] INFO o.a.i.d.c.IoTDBThreadPoolFactory:55 - new fixed thread pool: Query, thread number: 8
2022-01-04 09:46:37,227 [ClusterClient-1] INFO o.a.i.d.s.TSServiceImpl:821 - Cost: 42152 ms, sql is select state from root.hbisdt.ntCDAlZT limit 10
2022-01-04 09:47:02,479 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Data(172.168.3.197:9003): Failed to synchronize with the leader after 20004ms
2022-01-04 09:47:23,485 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Meta: Failed to synchronize with the leader after 20003ms
2022-01-04 09:47:44,493 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Meta: Failed to synchronize with the leader after 20004ms
2022-01-04 09:48:05,499 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Data(172.168.3.197:9003): Failed to synchronize with the leader after 20004ms
2022-01-04 09:48:05,504 [ClusterClient-1] INFO o.a.i.d.s.TSServiceImpl:821 - Cost: 42016 ms, sql is select state from root.hbisdt.ntCDAlZT limit 10
2022-01-04 09:48:10,262 [pool-2-IoTDB-timedQuerySqlCountThread-1] INFO o.a.i.d.s.TSServiceImpl:201 - Query count in current 1 minute 1
2022-01-04 09:49:03,564 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Data(172.168.3.197:9003): Failed to synchronize with the leader after 20004ms
2022-01-04 09:49:24,572 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Meta: Failed to synchronize with the leader after 20006ms
2022-01-04 09:49:45,579 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Meta: Failed to synchronize with the leader after 20004ms
2022-01-04 09:50:06,586 [ClusterClient-1] WARN o.a.i.c.s.m.RaftMember:950 - Data(172.168.3.197:9003): Failed to synchronize with the leader after 20005ms
2022-01-04 09:50:06,591 [ClusterClient-1] INFO o.a.i.d.s.TSServiceImpl:821 - Cost: 42017 ms, sql is select state from root.hbisdt.ntCDAlZT limit 10

超慢节点日志
[第三节点日志.zip](https://github.com/apache/iotdb/files/7805067/default.zip)
修改 3 个节点 conf/logback.xml 文件里面的 root level="info" 为 root level="debug"后清空日志文件,重启各个节点再次执行同样查询SQL,发现只有一个节点1快,另外两个成慢的了

三个节点的所有日志
[三节点日志.zip](https://github.com/apache/iotdb/files/7806247/default.zip)
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia dai log allegati dei tre nodi e dagli avvisi di RaftMember che mostrano timeout di sincronizzazione ripetuti, quindi confronta i tempi delle query e la sequenza di riavvio tra i nodi. Riproduci lo stesso SQL su ogni nodo ed esamina i log pertinenti del cluster e del servizio; il lavoro è completato quando viene identificato il motivo per cui i ritardi di sincronizzazione influenzano la latenza delle query e viene documentata una risoluzione verificata.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- databases, distributed-systems, performance
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 38/100