MANIFEST stops updating, but new SST files being written
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
The first sign something was wrong with our RocksDB instance (6.2.4) happened with our backup script: Checkpoint creation started failing with `Corruption: file too small` beginning from the 5th of February.
This was not noticed for 2 weeks, during which everything appeared to work normally: All key-value operations done by RocksDB were correct as far as we can tell.
Once we noticed however, we realized `CURRENT`, as well as the `MANIFEST` file it was pointing to had not been updated since the 5th of February, even though many new SST files had been written since then, and some deleted through compaction.
Even more strange, `LOG` had similarly not been appended to since the 5th of February.
The periodic thread running `VerifyChecksum` every day did not trigger a crash, meaning `VerifyChecksum` was either stuck / deadlocked too, or it was succeeding and reporting no errors.
Upon restart of the process, RocksDB would no longer open the DB, complaining about missing SST files. At that point, we had managed to recover the data through other means, and have not attempted to repair the corrupted RocksDB directory yet, but we've kept a copy of course.
We're still trying to get more information on what happened exactly, but I thought maybe `MANIFEST` and `LOG` not updating might be specific enough to ring any bells for you? Has this been observed before?
Unfortunately we had trouble running `pstack` to obtain a stacktrace right before stopping the process -- we may have a core file from `gcore` but its quality is dubious. If I manage to obtain more information from the corefile I'll add it here.
Thanks!
### Expected behavior
`MANIFEST` should keep getting updated when new SST files are written.
### Actual behavior
`MANIFEST` got stuck in a revision in the past, even though new SST files had been written, and some of the referenced files in the old `MANIFEST` had already been deleted through compaction.
### Steps to reproduce the behavior
N/A
Contributor guide
Assessment
This issue has not been assessed yet.