cockroachdb / cockroachdb/cockroach
storage: detect durability loss after restart
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Copied from https://github.com/cockroachdb/pebble/issues/4430#issuecomment-2781450143
Since disabling fsync isn’t safe for CRDB, it would be nice to detect situations when Pebble might have lost some durable writes as a result of it. This would significantly reduce investigation time for such issues. Sketch of the approach:
The idea is to have a “proof” that Pebble has been synced, or the system has been running continuously between consecutive Pebble starts and couldn't lose writes (with high probability).
To detect that the system has been running continuously, we need some kind of “epoch” that changes with every system start. A good candidate for this is the system boot time, which can be found with `grep btime /proc/stat` (or `sysctl kern.boottime` on MacOS).
Example on my machine:
```
$ sysctl kern.boottime
kern.boottime: { sec = 1742501182, usec = 290861 } Thu Mar 20 20:06:22 2025
When Pebble starts up, check for a `LAST_BOOT` file.
```
- If `LAST_BOOT` does not exist, this is a first start, or a clean shutdown had been performed. Create `LAST_BOOT` and put the current epoch in it.
- If `LAST_BOOT` exists, compare the epoch noted in that file with the current one. If they mismatch, that means the system restarted since the last time Pebble was running, and there hasn’t been a clean shutdown. There are options on how to handle it (could be configurable): either start with a loud message that there could be data loss, or refuse to start Pebble.
- On clean shutdown (after WAL is synced), remove `LAST_BOOT`.
- Operators can also force a clean shutdown by syncing Pebble directory / file system (e.g. run the `sync` command), and removing `LAST_BOOT`.
Jira issue: CRDB-53407
Contributor guide
Assessment
This issue has not been assessed yet.