[Bug] StorageEngine failed to recover 导致无法写入数据
- Lingua principale
- Java
- Stelle
- 6.4k
- Fork
- 1.2k
- Merge medio
- 1g 23h
- PR unite (30g)
- 115
Descrizione
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
- IoTDB:`0.13.3`
- OS:`CentOS Linux release 7.5.1804 (Core)`
- JDK:
```
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9
```
### Describe the bug and provide the minimal reproduce step
生产环境有一个节点不知道什么原因有可能是断电或物理机重启导致IoTDB在重启后,无法写入数据,查询部分正常。
执行Insert语句返回的错误信息:
```
Msg: 304: Path *** does not exist
```
重新启动iotdb服务之后:在日志中发现了以下信息:
- log_error.log
```
2023-03-10 14:37:05,903 [pool-12-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.t.r.TsFileSequenceReader:280 - Something error happened while reading file metadata of file /var/lib/iotdb/data/data/sequence/root.fz4.pv.s_9/0/0/1675621909201-1-0-32.tsfile
2023-03-10 14:37:05,949 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-28
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:402)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:400)
... 1 common frames omitted
Caused by: java.nio.BufferUnderflowException: null
at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:647)
at java.base/java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:439)
at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readLong(ReadWriteIOUtils.java:546)
at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:105)
at org.apache.iotdb.tsfile.file.metadata.TsFileMetadata.deserializeFrom(TsFileMetadata.java:55)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readFileMetadata(TsFileSequenceReader.java:275)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getAllTimeseriesMetadata(TsFileSequenceReader.java:935)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:183)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:781)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:519)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.(VirtualStorageGroupProcessor.java:402)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:777)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted
```
### What did you expect to see?
- 写入成功;
### What did you see instead?
- 请分析一下什么情况下造成此种现象?
- 如何规避?
- 出现之后如何解决?
- 是否可以直接删除 错误日志中 提到的那个文件?
- 如果删除这个文件之后会出现什么问题?
- 有没有建议好的策略能够减小该异常的影响范围?
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Iniziare dal percorso di ripristino indicato nello stack trace: StorageEngine.recover, VirtualStorageGroupProcessor.recoverTsFiles e TsFileRecoverPerformer.recoverResourceFromReader. Esaminare l’errore segnalato nella lettura dei metadati di TsFileSequenceReader e determinare il comportamento di ripristino previsto per il file interessato. Il lavoro è completato quando sono documentati la causa, la prevenzione o mitigazione e i passaggi sicuri per la risoluzione, incluse le conseguenze dell’eliminazione del file.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- databases
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100