[Bug] StorageEngine failed to recover 导致无法写入数据
- Langage dominant
- Java
- Étoiles
- 6.4k
- Forks
- 1.2k
- Merge moyen
- 1 j 23 h
- PR mergées (30 j)
- 115
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
- IoTDB:`0.13.3`
- OS:`CentOS Linux release 7.5.1804 (Core)`
- JDK:
```
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9
```
### Describe the bug and provide the minimal reproduce step
生产环境有一个节点不知道什么原因有可能是断电或物理机重启导致IoTDB在重启后,无法写入数据,查询部分正常。
执行Insert语句返回的错误信息:
```
Msg: 304: Path *** does not exist
```
重新启动iotdb服务之后:在日志中发现了以下信息:
- log_error.log
```
2023-03-10 14:37:05,903 [pool-12-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.t.r.TsFileSequenceReader:280 - Something error happened while reading file metadata of file /var/lib/iotdb/data/data/sequence/root.fz4.pv.s_9/0/0/1675621909201-1-0-32.tsfile
2023-03-10 14:37:05,949 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-28
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:402)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:400)
... 1 common frames omitted
Caused by: java.nio.BufferUnderflowException: null
at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:647)
at java.base/java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:439)
at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readLong(ReadWriteIOUtils.java:546)
at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:105)
at org.apache.iotdb.tsfile.file.metadata.TsFileMetadata.deserializeFrom(TsFileMetadata.java:55)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readFileMetadata(TsFileSequenceReader.java:275)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getAllTimeseriesMetadata(TsFileSequenceReader.java:935)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:183)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:781)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:519)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.(VirtualStorageGroupProcessor.java:402)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:777)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted
```
### What did you expect to see?
- 写入成功;
### What did you see instead?
- 请分析一下什么情况下造成此种现象?
- 如何规避?
- 出现之后如何解决?
- 是否可以直接删除 错误日志中 提到的那个文件?
- 如果删除这个文件之后会出现什么问题?
- 有没有建议好的策略能够减小该异常的影响范围?
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par le chemin de récupération nommé dans la stack trace : StorageEngine.recover, VirtualStorageGroupProcessor.recoverTsFiles et TsFileRecoverPerformer.recoverResourceFromReader. Examinez l’échec signalé lors de la lecture des métadonnées de TsFileSequenceReader et déterminez le comportement de récupération attendu pour le fichier concerné. La tâche est considérée comme terminée lorsque la cause, la prévention ou l’atténuation, ainsi que les étapes de remédiation sûres, sont documentées, y compris les conséquences de la suppression du fichier.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- databases
- Type d'issue
- Bug
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 25/100