apache / apache/iotdb

[Bug] StorageEngine failed to recover 导致无法写入数据

Aperta
#9,284 5 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
6.4k
Fork
1.2k
Merge medio
1g 23h
PR unite (30g)
115

Descrizione

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

- IoTDB:`0.13.3`
- OS:`CentOS Linux release 7.5.1804 (Core)`
- JDK:

```
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9

```

### Describe the bug and provide the minimal reproduce step

生产环境有一个节点不知道什么原因有可能是断电或物理机重启导致IoTDB在重启后,无法写入数据,查询部分正常。

执行Insert语句返回的错误信息:

```
Msg: 304: Path *** does not exist
```

重新启动iotdb服务之后:在日志中发现了以下信息:

- log_error.log
```

2023-03-10 14:37:05,903 [pool-12-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.t.r.TsFileSequenceReader:280 - Something error happened while reading file metadata of file /var/lib/iotdb/data/data/sequence/root.fz4.pv.s_9/0/0/1675621909201-1-0-32.tsfile
2023-03-10 14:37:05,949 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-28
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:402)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:400)
... 1 common frames omitted
Caused by: java.nio.BufferUnderflowException: null
at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:647)
at java.base/java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:439)
at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readLong(ReadWriteIOUtils.java:546)
at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:105)
at org.apache.iotdb.tsfile.file.metadata.TsFileMetadata.deserializeFrom(TsFileMetadata.java:55)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readFileMetadata(TsFileSequenceReader.java:275)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getAllTimeseriesMetadata(TsFileSequenceReader.java:935)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:183)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:781)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:519)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.(VirtualStorageGroupProcessor.java:402)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:777)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted

```

### What did you expect to see?

- 写入成功;

### What did you see instead?

- 请分析一下什么情况下造成此种现象?
- 如何规避?
- 出现之后如何解决?
- 是否可以直接删除 错误日志中 提到的那个文件?
- 如果删除这个文件之后会出现什么问题?
- 有没有建议好的策略能够减小该异常的影响范围?

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Iniziare dal percorso di ripristino indicato nello stack trace: StorageEngine.recover, VirtualStorageGroupProcessor.recoverTsFiles e TsFileRecoverPerformer.recoverResourceFromReader. Esaminare l’errore segnalato nella lettura dei metadati di TsFileSequenceReader e determinare il comportamento di ripristino previsto per il file interessato. Il lavoro è completato quando sono documentati la causa, la prevenzione o mitigazione e i passaggi sicuri per la risoluzione, incluse le conseguenze dell’eliminazione del file.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
databases
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.