cockroachdb / cockroachdb/pebble
db: pipeline WAL rotation
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
Let _L_ be the fsync latency of the WAL storage medium.
When the memtable and WAL are rotated, the first batch application to the new WAL may need to at worst wait:
1. For an inflight fsync of entries to the previous WAL to complete (at worst, _L_).
2. For a final fsync of entries to the previous WAL that did not make the in-flight fsync. ( _L_ )
3. A final fsync in `LogWriter.Close` to ensure the EOF trailer is synced. ( _L_ )
4. A fsync of the WAL directory to ensure the new WAL is durably linked into its new name. ( _L_ )
5. The fsync of this new batch. ( _L_ )
Cumulatively, these can cause commit tail latencies to increase 5x. There are a few ways this could be reduced.
(2) & (3) could be together bounded by 1 _L_ through more coordination between `LogWriter.Close` and the `LogWriter`'s flush loop. The final flush of log entries (2) can include the EOF trailer and sync:
https://github.com/cockroachdb/pebble/blob/f6eaf9a696e6344af4660b2ac7e30e70539ac2f5/record/log_writer.go#L638-L645
(4) & (5) could happen in parallel, but it would require some additional, delicate synchronization.
Or alternatively we could prepare the next WAL ahead of time. In a steady state, Pebble would have two open WALs with log numbers `>= minUnflushedLogNum`: _current_ and _next_. The _next_ LogWriter's `flushLoop` would synchronize with _current_'s `Close`, refusing to signal to waiting syncQueuers until _current_'s `Close` has completed. By addressing (2) & (3) as well, this would eliminate any additional worst-case fsync latency from the WAL rotation itself, making it inline with ordinary WAL fsyncs.
In `Open`, we would need to relax/rework the `strictWALTail` option. Currently all replayed WALs besides the most recent one are required to have clean tails indicating that they were deliberately closed—anything else is interpreted as corruption. With this change, it would be possible for the second most recent WAL to have an unclean tail for some time. We could include a marker entry in the _next_ WAL that is written only once after the _next_ WAL observed that _current_'s `Close` completed, indicating that if recovery observed an unclean tail of the previous WAL, it should treat it as corruption.
Jira issue: PEBBLE-192
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with record/log_writer.go around lines 638-645, then trace LogWriter.Close, its flush loop, and Open's strictWALTail handling. Compare the proposed WAL pre-creation and synchronization approaches, and define completion as reducing rotation-induced fsync latency while preserving correct recovery behavior for unclean WAL tails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- database, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100