cockroachdb / cockroachdb/cockroach

storage: detect durability loss after restart

Open
#151,666 0 comments 0 reactions 0 assignees View on GitHub
A-storage C-enhancement E-starter T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Copied from https://github.com/cockroachdb/pebble/issues/4430#issuecomment-2781450143

Since disabling fsync isn’t safe for CRDB, it would be nice to detect situations when Pebble might have lost some durable writes as a result of it. This would significantly reduce investigation time for such issues. Sketch of the approach:

The idea is to have a “proof” that Pebble has been synced, or the system has been running continuously between consecutive Pebble starts and couldn't lose writes (with high probability).

To detect that the system has been running continuously, we need some kind of “epoch” that changes with every system start. A good candidate for this is the system boot time, which can be found with `grep btime /proc/stat` (or `sysctl kern.boottime` on MacOS).

Example on my machine:

```
$ sysctl kern.boottime
kern.boottime: { sec = 1742501182, usec = 290861 } Thu Mar 20 20:06:22 2025
When Pebble starts up, check for a `LAST_BOOT` file.
```

- If `LAST_BOOT` does not exist, this is a first start, or a clean shutdown had been performed. Create `LAST_BOOT` and put the current epoch in it.
- If `LAST_BOOT` exists, compare the epoch noted in that file with the current one. If they mismatch, that means the system restarted since the last time Pebble was running, and there hasn’t been a clean shutdown. There are options on how to handle it (could be configurable): either start with a loud message that there could be data loss, or refuse to start Pebble.
- On clean shutdown (after WAL is synced), remove `LAST_BOOT`.
- Operators can also force a clean shutdown by syncing Pebble directory / file system (e.g. run the `sync` command), and removing `LAST_BOOT`.

Jira issue: CRDB-53407

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.