apache / apache/iotdb

[Bug] StorageEngine failed to recover 导致无法写入数据

Open
#9,284 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

- IoTDB:`0.13.3`
- OS:`CentOS Linux release 7.5.1804 (Core)`
- JDK:

```
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9

```

### Describe the bug and provide the minimal reproduce step

生产环境有一个节点不知道什么原因有可能是断电或物理机重启导致IoTDB在重启后,无法写入数据,查询部分正常。

执行Insert语句返回的错误信息:

```
Msg: 304: Path *** does not exist
```

重新启动iotdb服务之后:在日志中发现了以下信息:

- log_error.log
```

2023-03-10 14:37:05,903 [pool-12-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.t.r.TsFileSequenceReader:280 - Something error happened while reading file metadata of file /var/lib/iotdb/data/data/sequence/root.fz4.pv.s_9/0/0/1675621909201-1-0-32.tsfile
2023-03-10 14:37:05,949 [Thread-1] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-1-28
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:402)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:400)
... 1 common frames omitted
Caused by: java.nio.BufferUnderflowException: null
at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:647)
at java.base/java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:439)
at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readLong(ReadWriteIOUtils.java:546)
at org.apache.iotdb.tsfile.file.metadata.MetadataIndexNode.deserializeFrom(MetadataIndexNode.java:105)
at org.apache.iotdb.tsfile.file.metadata.TsFileMetadata.deserializeFrom(TsFileMetadata.java:55)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readFileMetadata(TsFileSequenceReader.java:275)
at org.apache.iotdb.tsfile.read.TsFileSequenceReader.getAllTimeseriesMetadata(TsFileSequenceReader.java:935)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromReader(TsFileRecoverPerformer.java:183)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recoverResourceFromFile(TsFileRecoverPerformer.java:175)
at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:108)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recoverTsFiles(VirtualStorageGroupProcessor.java:781)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.recover(VirtualStorageGroupProcessor.java:519)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.(VirtualStorageGroupProcessor.java:402)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:777)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted

```

### What did you expect to see?

- 写入成功;

### What did you see instead?

- 请分析一下什么情况下造成此种现象?
- 如何规避?
- 出现之后如何解决?
- 是否可以直接删除 错误日志中 提到的那个文件?
- 如果删除这个文件之后会出现什么问题?
- 有没有建议好的策略能够减小该异常的影响范围?

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the recovery path named in the stack trace: StorageEngine.recover, VirtualStorageGroupProcessor.recoverTsFiles, and TsFileRecoverPerformer.recoverResourceFromReader. Inspect the reported TsFileSequenceReader metadata-read failure and determine the expected recovery behavior for the affected file. Done means documenting the cause, prevention or mitigation, and safe remediation steps, including the consequences of deleting the file.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.